You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by steveloughran <gi...@git.apache.org> on 2015/04/08 15:52:12 UTC
[GitHub] spark pull request: SPARK-1537 Application Timeline Server integra...
GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/5423
SPARK-1537 Application Timeline Server integration
This a snapshot of the work in progress. It's a superset of zhzhan's work, compiling against the master branch and with a lot more tests.
1. there's now a hadoop-2.6 profile in the root pom; this is used to include the yarn/history source tree containing the new sources and tests
1. As well as Marcelo's mock tests (which are all working happily), there's now some functional tests which use the timeline client to marshall a spark event, push it to an in-VM timeline server, then verify it's there/unmarshallable.
1. There's some improved robustness in the production-side code.
This is not complete; i've pushed it up to show the current status.
What's needed now is the REST operations to retrieve data, tests that everything wired up works.
## Production Code
1. I'm going to add a Jersey client to talk to ATS, avoiding any use of private hadoop classes, and resulting in code that works with Hadoop 2.6+
1. I think the code to do PUT operations needs to be made async, so transient ATS outages don't block things. It's already queued & running in its own thread, but its posting in a synchronized section.
## Tests
* listening to spark context -> posted events in the ATS
* retrieving events from the ATS via GET
* joined up history provider
## Hadoop side
A statement of the compatibility policy of the current ATS read and write REST API
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/steveloughran/spark stevel/feature/SPARK-1537-ATS
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5423.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5423
----
commit 8b8f0a80e62eb8e2647467b2daac82b758a5fb10
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-01T14:18:30Z
SPARK-1537 move 1.2.1 version of ATS support into branch-1.4 code, adding a hadoop-2.6 profile in the process, for conditional load of yarn/history sources
commit 4d92bf12730c8b0964ebf45e04b3d337adec8353
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-01T14:28:37Z
SPARK-1537 milestone: everything compiles
commit 010211c801cc1aa21174724c3a13d6a2dbbcb5d9
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-02T18:17:10Z
SPARK-1537 interim commit of ongoing test work
commit d8ae297d4668ec9eaee06fdc08dfb32cffc5987b
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-02T19:45:40Z
SPARK-1537 slowly improving test success rate
commit 957d1c191bdd6498d5416926b41069f734795fcb
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-07T11:31:09Z
SPARK-1537 add initial non-mock implementation
commit 56d3146d18e29f66f95ac19a8379ec7b68090a42
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-07T13:26:35Z
SPARK-1537 TimeineOperationsSuite with abstract base class
commit 94291ccf71f2849886db6c3d670995d08e079c5b
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-07T15:14:51Z
SPARK-1537 Marshalling tests & some better handling of (basic) parse problems
commit 49913a2e7379b5b437bb6bf61997639dcd5a27d9
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-07T16:33:11Z
SPARK-1537 Marshall/unmarshall is not working locally (assumption, needed to be sent over as JSON first)
commit 61c1d593d53d659211db1dae7488b6466d600698
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-07T17:34:06Z
SPARK-1537 Marshall/unmarshall is not working locally (assumption, needed to be sent over as JSON first)
commit 008aa4eb0d2d1ab106d1d07063d0bc39ed9a5e92
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-07T19:01:49Z
SPARK-1537 marshalling working, the limited set of tests are good
commit e595ae33830e2fbdb042cb3a79f7686822736400
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-08T12:02:49Z
SPARK-1537 factoring out common parts of the test into a set of traits
commit 8fdd8ffc977c894643feb740ca6406a80d60d32d
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-08T13:18:43Z
SPARK-1537 round trip tests from timeline client to ATS & back working
commit 1543f47ad06c218f818cd4fa6bb88f2fea91607d
Author: Steve Loughran <st...@hortonworks.com>
Date: 2015-04-08T13:19:42Z
SPARK-1537 round trip tests from timeline client to ATS & back working
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28168907
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala ---
@@ -56,10 +59,16 @@ private[spark] class YarnClientSchedulerBackend(
totalExpectedExecutors = args.numExecutors
client = new Client(args, conf)
appId = client.submitApplication()
+ services.start(sc, appId)
+ services.addShutdownHook(ApplicationMaster.SHUTDOWN_HOOK_PRIORITY - 1)
waitForApplication()
asyncMonitorApplication()
}
+ override def stopExecutors(): Unit = {
+ super.stopExecutors()
+ }
+
--- End diff --
Why need this, since it only call the same parent method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162571699
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47267/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-142672096
[Test build #42913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42913/consoleFull) for PR 5423 at commit [`da6ab94`](https://github.com/apache/spark/commit/da6ab94327028e82040c934504ed1ac0e6e17170).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-152566557
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44685/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147823047
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43649/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139051560
[Test build #42213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42213/console) for PR 5423 at commit [`c16f112`](https://github.com/apache/spark/commit/c16f1129336c5383c8e0bd62dcdb3f6425a5bfde).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnExtensionService extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94583088
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30607/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-140049764
[Test build #42423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42423/consoleFull) for PR 5423 at commit [`7869bea`](https://github.com/apache/spark/commit/7869bea713ab04811004062d2ed324371f96597f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-152535987
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-142682960
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42913/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151825769
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149602450
**[Test build #43979 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43979/consoleFull)** for PR 5423 at commit [`a0666b5`](https://github.com/apache/spark/commit/a0666b5acab030b9a3de154dd1acd23d75bf5d16).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `trait SchedulerExtensionService `\n * `case class SchedulerExtensionServiceBinding(`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-146959330
Jerry, the code is where it is to isolate it across versions; it started off where it was in the branch 1.2 era, where the split was common, alpha and stable
Having another pom would be the maven way, it would avoid the tweaks needed to expand the source and test files (and one not pushed up to include the dirs in checkstyle -which was why style issues weren't being picked up by it).
with `yarn/src/history/`, there's the complexity of production and test sources, so things in the source tree would still be messy, you'd have with
```
yarn/src/history/main/scala
yarn/src/history/test/scala
yarn/src/history/test/resources
```
so...it's still all source, we'd just have a whole extra tree in there. Not significantly cleaner and we'd still need to have some bits in the POM to deal with it.
There's one other variant, which is have it in as `yarn/src/hadoop-2.6/...` . That is no different from `yarn/src/history/`, except that if there is any more hadoop 2.6-specific YARN code (after all, it's only that dependency that is keeping it out the main tree), then it could go in the same place. There are bits in Hadoop 2.6 that haven't been picked up on yet -this would be the place to add that stuff in isolation.
If people think that's the best name, how about I isolate that change into a separate patch, one with the pom changes and the diffs to the existing yarn/src which adds the new yarn extension service trait —something @vanzin suggested I do anyway? Then that can go in as a first step for this bigger patch?
(Whatever happens, I can see I'll be enjoying handling big changes across parallel branches. But it's good to get the layout right before its out in the wild)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147770129
(this probably isn't going to compile BTW)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151260664
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147777894
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43645/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95707645
[Test build #30862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30862/consoleFull) for PR 5423 at commit [`957daf5`](https://github.com/apache/spark/commit/957daf5ccfade15cd45bf0cc56f4a02c43f1f3e2).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnService extends Closeable `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-145978872
[Test build #43296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43296/consoleFull) for PR 5423 at commit [`70491a9`](https://github.com/apache/spark/commit/70491a9d849bf6f0ffe39c57dea2f1fdceff7855).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-97243945
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149602649
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162602535
**[Test build #47270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47270/consoleFull)** for PR 5423 at commit [`0adb70a`](https://github.com/apache/spark/commit/0adb70a1240ea4f6fb7f5f78fc54d2facf7ddbef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28507400
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala ---
@@ -56,10 +59,16 @@ private[spark] class YarnClientSchedulerBackend(
totalExpectedExecutors = args.numExecutors
client = new Client(args, conf)
appId = client.submitApplication()
+ services.start(sc, appId)
+ services.addShutdownHook(ApplicationMaster.SHUTDOWN_HOOK_PRIORITY - 1)
waitForApplication()
asyncMonitorApplication()
}
+ override def stopExecutors(): Unit = {
+ super.stopExecutors()
+ }
+
--- End diff --
well-caught, probably an override that I ended up not expanding
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147786473
[Test build #43649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43649/consoleFull) for PR 5423 at commit [`53a56cd`](https://github.com/apache/spark/commit/53a56cdc4658d4fe0bd81ef4a95620421da9034b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-138996188
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139051703
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-142670528
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-142670550
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-137838190
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-140048755
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-137788331
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151934240
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147777756
[Test build #43645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43645/console) for PR 5423 at commit [`13259d8`](https://github.com/apache/spark/commit/13259d892555e5339275f37a97818c1d1f4a3a79).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait SchedulerExtensionService `
* `case class SchedulerExtensionServiceBinding(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94760699
[Test build #30669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30669/consoleFull) for PR 5423 at commit [`0d29785`](https://github.com/apache/spark/commit/0d2978556307ae6cc9fb990c6f358eee87457a3c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-137789617
[Test build #42005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42005/consoleFull) for PR 5423 at commit [`7a348f5`](https://github.com/apache/spark/commit/7a348f553b6b747d76ceb7f4e51478f875df36b0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-143021428
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94949299
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30695/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151260491
**[Test build #44370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44370/consoleFull)** for PR 5423 at commit [`59c50cc`](https://github.com/apache/spark/commit/59c50cc719b1efe0c395a39bf33d55ebcc328a6b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149563266
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-142682957
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162571693
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95328939
[Test build #30780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30780/consoleFull) for PR 5423 at commit [`c8e73e0`](https://github.com/apache/spark/commit/c8e73e06cf3a719d964835df887005dd49d611ec).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnService extends Closeable `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162571320
**[Test build #47267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47267/consoleFull)** for PR 5423 at commit [`6dac1bb`](https://github.com/apache/spark/commit/6dac1bb1d48b89a0bab9facfba95e73061f0f2a3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147783246
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-143021447
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149578694
(this is just a rebase + setup for splitting the initial cluster scheduler extension into its own pull-request)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-107662797
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94945069
[Test build #30695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30695/consoleFull) for PR 5423 at commit [`4c0dd85`](https://github.com/apache/spark/commit/4c0dd85d0e3916c10f756fcc33e075924eddce58).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r38831693
--- Diff: docs/monitoring.md ---
@@ -256,6 +255,247 @@ still required, though there is only one application available. Eg. to see the
running app, you would go to `http://localhost:4040/api/v1/applications/[app-id]/jobs`. This is to
keep the paths consistent in both modes.
+## Hadoop YARN Timeline service history provider
+
+As well as the Filesystem History Provider, Spark can integrate with the Hadoop YARN
+"Application Timeline Service". This is a service which runs in a YARN cluster, recording
+application- and YARN- published events to a database, retrieving them on request.
+
+Spark integrates with the timeline service by
+1. Publishing events to the timeline service as applications execute.
+1. Listing application histories published to the timeline service.
+1. Retrieving the details of specific application histories.
+
+### Configuring the Timeline Service
+
+For details on configuring and starting the timeline service, consult the Hadoop documentation.
+
+From the perspective of Spark, the key requirements are
+1. The YARN timeline service must be running.
+1. Its URL is known, and configured in the `yarn-site.xml` configuration file.
+1. The user has an Kerberos credentials required to interact with the service.
+
+The timeline service URL must be declared in the property `yarn.timeline-service.webapp.address`,
+or, if HTTPS is the protocol, `yarn.timeline-service.webapp.https.address`
+
+The choice between HTTP and HTTPS is made on the value of `yarn.http.policy`, with can be one of
--- End diff --
typo `Which`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-140115372
The publisher/yarn scheduler side of this code is now in a separate pull request, [8744](https://github.com/apache/spark/pull/8744); this is the aggregate request containing that code and the history server side
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-140105103
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-164061802
**[Test build #47596 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47596/consoleFull)** for PR 5423 at commit [`e134e29`](https://github.com/apache/spark/commit/e134e29c8ae121dbacb9e09c70e0c4820915c296).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-97197522
[Test build #31164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31164/consoleFull) for PR 5423 at commit [`0f6860b`](https://github.com/apache/spark/commit/0f6860b64c3f9e458f8f9ece692828c1d5129f38).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94573736
[Test build #30605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30605/consoleFull) for PR 5423 at commit [`0f0f66c`](https://github.com/apache/spark/commit/0f0f66c810a0ea3c647ddf70e73c2a79e932d2f7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-163644632
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47495/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-161436970
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47081/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162998150
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162602714
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-137838083
[Test build #42005 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42005/console) for PR 5423 at commit [`7a348f5`](https://github.com/apache/spark/commit/7a348f553b6b747d76ceb7f4e51478f875df36b0).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnExtensionService extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-140105106
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42423/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95282650
This iteration has a simpler service flush/shutdown logic, with specific messages for each action queued, and no attempt to trigger the yarn service stop when a stopApplication event is received.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94805435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30669/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-93580786
[Test build #30378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30378/consoleFull) for PR 5423 at commit [`8446042`](https://github.com/apache/spark/commit/8446042b62369382b312dedcad047b27af3ccd54).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162602716
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47270/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151876039
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151975095
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44539/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94805373
[Test build #30669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30669/consoleFull) for PR 5423 at commit [`0d29785`](https://github.com/apache/spark/commit/0d2978556307ae6cc9fb990c6f358eee87457a3c).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnService extends Closeable `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95350787
**[Test build #30781 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30781/consoleFull)** for PR 5423 at commit [`957daf5`](https://github.com/apache/spark/commit/957daf5ccfade15cd45bf0cc56f4a02c43f1f3e2) after a configured wait of `150m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-91605343
[Test build #30029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30029/consoleFull) for PR 5423 at commit [`ba2e0a9`](https://github.com/apache/spark/commit/ba2e0a9a1f8d8c307a6358113b2a5f80a04b1165).
* This patch **passes all tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `case class TimestampEvent(sparkEvent: SparkListenerEvent, time: Long, flush: Boolean = false) `
* `class YarnEventListener(sc: SparkContext, service: YarnHistoryService)`
* `class YarnHistoryProvider(sparkConf: SparkConf)`
* `class YarnHistoryService extends AbstractService("History Service")`
* `trait YarnService extends Closeable `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-97196771
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151827062
**[Test build #44526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44526/consoleFull)** for PR 5423 at commit [`2c1db93`](https://github.com/apache/spark/commit/2c1db93bb1fe72a03e4b866741b6b803b30bb2b3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149568101
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43976/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-137788310
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-156398346
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-156432660
**[Test build #45859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45859/consoleFull)** for PR 5423 at commit [`ccfeba7`](https://github.com/apache/spark/commit/ccfeba704611a78cf27f77f78abfae420c45da42).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-168112695
I'm going to close this pull request. If this is still relevant and you are interested in pushing it forward, please open a new pull request. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151933545
.. rebase in sync with (and containing) publisher and service extension patches ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94583087
[Test build #30607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30607/consoleFull) for PR 5423 at commit [`1256532`](https://github.com/apache/spark/commit/1256532b65c27196afb5a40ea33151f8d269a05e).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class TimestampEvent(sparkEvent: SparkListenerEvent, time: Long, flush: Boolean = false) `
* `class YarnHistoryProvider(sparkConf: SparkConf)`
* `trait YarnService extends Closeable `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147749606
[Test build #43645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43645/consoleFull) for PR 5423 at commit [`13259d8`](https://github.com/apache/spark/commit/13259d892555e5339275f37a97818c1d1f4a3a79).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-163800395
**[Test build #47550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47550/consoleFull)** for PR 5423 at commit [`20514a7`](https://github.com/apache/spark/commit/20514a75c7484a81aa2f7df1bb9ff8b54cb89651).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139331304
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95676102
[Test build #30862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30862/consoleFull) for PR 5423 at commit [`957daf5`](https://github.com/apache/spark/commit/957daf5ccfade15cd45bf0cc56f4a02c43f1f3e2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-138998061
[Test build #42212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42212/consoleFull) for PR 5423 at commit [`1736bb6`](https://github.com/apache/spark/commit/1736bb68cb5b0676d38c208a4255924618682687).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151876042
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44526/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-107662816
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28468215
--- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala ---
@@ -0,0 +1,630 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history.yarn
+
+import java.net.{ConnectException, URL}
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger}
+
+import scala.collection.mutable.LinkedList
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.service.{AbstractService, Service}
+import org.apache.hadoop.yarn.api.records.ApplicationId
+import org.apache.hadoop.yarn.api.records.timeline.{TimelineDomain, TimelineEntity, TimelinePutResponse}
+import org.apache.hadoop.yarn.client.api.TimelineClient
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+import org.apache.hadoop.yarn.exceptions.YarnException
+
+import org.apache.spark.deploy.history.yarn.YarnTimelineUtils._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster.YarnService
+import org.apache.spark.{Logging, SparkContext}
+
+/**
+ * Implements a Hadoop service with the init/start logic replaced by that
+ * of the YarnService.
+ * <p>
+ * As <code>AbstractService</code> implements <code>close()</code>, routing
+ * to its <code>stop</code> method, calling <code>close()</code> is sufficient
+ * to stop the service instance.
+ * <p>
+ * However, when registered to receive spark events, the service will continue to
+ * receive them until the spark context is stopped. Events received when this service
+ * is in a <code>STOPPED</code> state will be discarded.
+ */
+private[spark] class YarnHistoryService extends AbstractService("History Service")
+ with YarnService with Logging {
+
+ private var sc: SparkContext = _
+ private var appId: ApplicationId = _
+ private var timelineClient: Option[TimelineClient] = None
+ private var listener: YarnEventListener = _
+ private var appName: String = null
+ private var userName: String = null
+ private var startTime: Long = _
+
+ private var batchSize: Int = YarnHistoryService.DEFAULT_BATCH_SIZE
+
+ // enqueue event to avoid blocking on main thread.
+ private val eventQueue = new LinkedBlockingQueue[TimestampEvent]
+ // cache layer to handle timeline client failure.
+ private var entityList = new LinkedList[TimelineEntity]
+ private var curEntity: Option[TimelineEntity] = None
+ // Do we have enough information filled for the entity
+ private var bAppStart = false
+ private var bAppEnd = false
+ // How many event we saved
+ private var curEventNum = 0
+ private val eventsProcessed: AtomicInteger = new AtomicInteger(0)
+ private val eventsQueued: AtomicInteger = new AtomicInteger(0)
+ private val eventPostFailures: AtomicInteger = new AtomicInteger(0)
+ private val flushCount = new AtomicInteger(0)
+ private var eventHandlingThread: Thread = null
+ private val stopped: AtomicBoolean = new AtomicBoolean(true)
+ private final val lock: AnyRef = new AnyRef
+ private var maxTimeToWaitOnShutdown: Long = YarnHistoryService.SHUTDOWN_WAIT_TIME
+ private var clientFailure = 0
+ private var domainId: String = null
+ private var timelineWebappAddress: URL = _
+
+
+ /**
+ * Create a timeline client
+ * @return the timeline client
+ */
+ def createTimelineClient = {
+ clientFailure += 1
+ logInfo("Creating timelineClient " + clientFailure)
+ val client = TimelineClient.createTimelineClient()
+ client.init(sc.hadoopConfiguration)
+ client.start
+ timelineClient = Some(client)
+ client
+ }
+
+ /**
+ * Get the timeline client; this will create it if needed
+ * @return the client
+ */
+ def getTimelineClient: TimelineClient = {
+ timelineClient.getOrElse(createTimelineClient)
+ }
+
+ def getEventsProcessed: Int = {
+ eventsProcessed.get()
+ }
+
+ def getEventsQueued: Int = {
+ eventsQueued.get()
+ }
+
+ def getQueueSize: Int = {
+ eventQueue.size()
+ }
+
+ def getBatchSize: Int = {
+ batchSize
+ }
+
+ def getEventPostFailures: Int = {
+ eventPostFailures.get()
+ }
+
+ /**
+ * Reset the timeline client
+ * <p>
+ * 1. Stop the timeline client service if running.
+ * 2. set the <code>timelineClient</code> field to <code>None</code>
+ */
+ def resetTimelineClient(): Unit = {
+ stopOptionalService(timelineClient)
+ timelineClient = None
+ }
+
+ /**
+ * Split a comma separated String, filter out any empty items, and return a Set of strings
+ */
+ private def stringToSet(list: String): Set[String] = {
+ list.split(',').map(_.trim).filter(!_.isEmpty).toSet
+ }
+
+ private def createTimelineDomain(): String = {
+ val sparkConf = sc.getConf
+ val aclsOn = sparkConf.getOption("spark.acls.enable").getOrElse(
+ sparkConf.get("spark.ui.acls.enable", "false")).toBoolean
+ if (!aclsOn) {
+ return null
+ }
+ val predefDomain = sparkConf.getOption("spark.ui.domain")
+ if (predefDomain.isDefined) {
+ domainId = predefDomain.get
+ return null
+ }
+ val current = UserGroupInformation.getCurrentUser.getShortUserName
+ val adminAcls = stringToSet(sparkConf.get("spark.admin.acls", ""))
+ val viewAcls = stringToSet(sparkConf.get("spark.ui.view.acls", ""))
+ val modifyAcls = stringToSet(sparkConf.get("spark.modify.acls", ""))
+
+ val readers = (adminAcls ++ modifyAcls ++ viewAcls).foldLeft(current)(_ + " " + _)
+ val writers = (adminAcls ++ modifyAcls).foldLeft(current)(_ + " " + _)
+ var tmpId = YarnHistoryService.DOMAIN_ID_PREFIX + appId
+ logInfo("Creating domain " + tmpId + " with readers: "
+ + readers + " and writers:" + writers)
+ val timelineDomain = new TimelineDomain()
+ timelineDomain.setId(tmpId)
+
+ timelineDomain.setReaders(readers)
+ timelineDomain.setWriters(writers)
+ try {
+ getTimelineClient.putDomain(timelineDomain)
+ } catch {
+ case e: YarnException => {
+ logError("cannot create the domain")
+ // fallback to default
+ tmpId = null
+ }
+ }
+ tmpId
+ }
+
+ def start(context: SparkContext, id: ApplicationId): Boolean = {
+ sc = context
+ appId = id
+ val yarnConf = new YarnConfiguration(context.hadoopConfiguration)
+ // the init() operation checks the state machine & prevents invocation out of sequence
+ init(yarnConf)
+ batchSize = sc.conf.getInt(YarnHistoryService.BATCH_SIZE, batchSize)
+
+ start()
+ if (yarnConf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ true
+ } else {
+ logInfo("Yarn timeline service not available, disabling client.")
+ false
+ }
+ }
+
+ override protected def serviceStart {
+ super.serviceStart()
+ val conf: Configuration = getConfig
+ if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ timelineWebappAddress = timelineWebappUri(conf).toURL
+ createTimelineClient
+ domainId = createTimelineDomain
+ eventHandlingThread = new Thread(new Dequeue(), "HistoryEventHandlingThread")
+ eventHandlingThread.start
+ } else {
+ logInfo("Yarn timeline service not available")
+ }
+ // irrespective of state, hook up to the listener
+ registerListener
+ logInfo(s"$this")
+ }
+
+ /**
+ * Return a summary of the service state to help diagnose problems
+ * during test runs, possibly even production
+ * @return a summary of the current service state
+ */
+ override def toString: String = {
+ super.toString +
+ s" endpoint=$timelineWebappAddress" +
+ s" bonded to ATS=$bondedToATS" +
+ s" listening=$listening" +
+ s" batchSize=$batchSize" +
+ s" flush count=$getFlushCount()" +
+ s" queue size=$getEventsQueued processed=$getEventsProcessed" +
+ s" post failures = $getEventPostFailures"
+ }
+
+ def listening: Boolean = {
+ listener != null;
+ }
+
+ /**
+ * Is the service hooked up to an ATS server. This does not
+ * check the validity of the link, only whether or not the service
+ * has been set up to talk to ATS.
+ * @return true if the service has a timeline client
+ */
+ def bondedToATS: Boolean = {
+ timelineClient != None;
+ }
+
+ /**
+ * Add the listener if it is not disabled.
+ * This is accessible in the same package purely for testing
+ * @return true if the register was enabled
+ */
+ private [yarn] def registerListener: Boolean = {
+ assert(sc != null, "Null context")
+ if (sc.conf.getBoolean(YarnHistoryService.REGISTER_LISTENER, true)) {
+ log.debug("Registering listener to spark context")
+ listener = new YarnEventListener(sc, this)
+ sc.listenerBus.addListener(listener)
+ true
+ } else {
+ log.info("Not listening to spark context events")
--- End diff --
logInfo
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139374411
[Test build #42278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42278/console) for PR 5423 at commit [`5c259d2`](https://github.com/apache/spark/commit/5c259d26ca98c1645ab31f9597df2f237361a90c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnExtensionService extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94578270
[Test build #30607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30607/consoleFull) for PR 5423 at commit [`1256532`](https://github.com/apache/spark/commit/1256532b65c27196afb5a40ea33151f8d269a05e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-156432776
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45859/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95675561
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r39079335
--- Diff: docs/monitoring.md ---
@@ -256,6 +255,247 @@ still required, though there is only one application available. Eg. to see the
running app, you would go to `http://localhost:4040/api/v1/applications/[app-id]/jobs`. This is to
keep the paths consistent in both modes.
+## Hadoop YARN Timeline service history provider
+
+As well as the Filesystem History Provider, Spark can integrate with the Hadoop YARN
+"Application Timeline Service". This is a service which runs in a YARN cluster, recording
+application- and YARN- published events to a database, retrieving them on request.
+
+Spark integrates with the timeline service by
+1. Publishing events to the timeline service as applications execute.
+1. Listing application histories published to the timeline service.
+1. Retrieving the details of specific application histories.
+
+### Configuring the Timeline Service
+
+For details on configuring and starting the timeline service, consult the Hadoop documentation.
+
+From the perspective of Spark, the key requirements are
+1. The YARN timeline service must be running.
+1. Its URL is known, and configured in the `yarn-site.xml` configuration file.
+1. The user has an Kerberos credentials required to interact with the service.
+
+The timeline service URL must be declared in the property `yarn.timeline-service.webapp.address`,
+or, if HTTPS is the protocol, `yarn.timeline-service.webapp.https.address`
+
+The choice between HTTP and HTTPS is made on the value of `yarn.http.policy`, with can be one of
--- End diff --
oops. fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151221247
**[Test build #44370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44370/consoleFull)** for PR 5423 at commit [`59c50cc`](https://github.com/apache/spark/commit/59c50cc719b1efe0c395a39bf33d55ebcc328a6b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-137838191
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42005/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151937749
**[Test build #44539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44539/consoleFull)** for PR 5423 at commit [`6057521`](https://github.com/apache/spark/commit/6057521fd1e14d7ac5eb60fd5650b37c0d2dd2d5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151975092
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94593553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30605/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-93605389
**[Test build #30378 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30378/consoleFull)** for PR 5423 at commit [`8446042`](https://github.com/apache/spark/commit/8446042b62369382b312dedcad047b27af3ccd54) after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5423
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-168316640
@steveloughran I was under the impression this was not meant to be merged, as it would require YARN 2.6 (2.7?) and that's not yet assume-able in Spark. At this point Spark 2.x is on 2.2+, but here's an argument maybe for bumping that up. But I do agree that long-lived PRs probably aren't ideal here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139331353
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-161436804
**[Test build #47081 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47081/consoleFull)** for PR 5423 at commit [`0b89edc`](https://github.com/apache/spark/commit/0b89edcaa00461138861695de7d5085b781d9a7d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-143313000
Hi guys, current folder structure under yarn is:
```
yarn ---
history ---
src ---
main ---
test ---
src ---
main ---
test ---
```
From my point this structure may not be so clear, I think there're several solutions:
1. Move `history` to `src` to keep consistent like other modules.
2. Or separate `history` from current `src` with different pom file, like what `sql` did. Also rename the `src`
That will be much clean IMHO, what's your opinion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-91569269
[Test build #30029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30029/consoleFull) for PR 5423 at commit [`ba2e0a9`](https://github.com/apache/spark/commit/ba2e0a9a1f8d8c307a6358113b2a5f80a04b1165).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95675502
Yes, it says it timed out (two comments up)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-164084775
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-93605396
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30378/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-156398325
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-152535962
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147777891
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162537898
**[Test build #47267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47267/consoleFull)** for PR 5423 at commit [`6dac1bb`](https://github.com/apache/spark/commit/6dac1bb1d48b89a0bab9facfba95e73061f0f2a3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-163800463
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-97243930
[Test build #31164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31164/consoleFull) for PR 5423 at commit [`0f6860b`](https://github.com/apache/spark/commit/0f6860b64c3f9e458f8f9ece692828c1d5129f38).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnService extends Closeable `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95328958
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30780/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95298856
[Test build #30780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30780/consoleFull) for PR 5423 at commit [`c8e73e0`](https://github.com/apache/spark/commit/c8e73e06cf3a719d964835df887005dd49d611ec).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149602654
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43979/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139515815
failing tests are in `BroadcastSuite`; not a codepath this patch goes near
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149563292
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-143057363
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42977/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-156432774
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147822907
[Test build #43649 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43649/console) for PR 5423 at commit [`53a56cd`](https://github.com/apache/spark/commit/53a56cdc4658d4fe0bd81ef4a95620421da9034b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait SchedulerExtensionService `
* `case class SchedulerExtensionServiceBinding(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-140104935
[Test build #42423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42423/console) for PR 5423 at commit [`7869bea`](https://github.com/apache/spark/commit/7869bea713ab04811004062d2ed324371f96597f).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnExtensionService extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-168317338
I'm about to resubmit it. The way the code is structured, the 2.6 specific stuff lives under yarn/src/history, as discussed in earlier points in this PR. Everything happily builds and tests on Hadoop <2.6, this feature and its tests only only get built on 2.6+
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-161436969
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-163785442
**[Test build #47550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47550/consoleFull)** for PR 5423 at commit [`20514a7`](https://github.com/apache/spark/commit/20514a75c7484a81aa2f7df1bb9ff8b54cb89651).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-146022351
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43296/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151220055
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-146022242
[Test build #43296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43296/console) for PR 5423 at commit [`70491a9`](https://github.com/apache/spark/commit/70491a9d849bf6f0ffe39c57dea2f1fdceff7855).
* This patch **passes all tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `trait YarnExtensionService `
* `case class YarnExtensionServiceBinding(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149564552
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162963733
**[Test build #47346 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47346/consoleFull)** for PR 5423 at commit [`b0e25bd`](https://github.com/apache/spark/commit/b0e25bdbcf16456791dd908fb2508dc0a22ad9db).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-163800464
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47550/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139051705
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42213/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149568390
**[Test build #43979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43979/consoleFull)** for PR 5423 at commit [`a0666b5`](https://github.com/apache/spark/commit/a0666b5acab030b9a3de154dd1acd23d75bf5d16).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-93582949
This is WiP build, with a lot more tests, with integration ones going all the way from a wired up spark context to an in-memory ATS server; this needs to be wrapped up with the GET calls to retrieve the data and verify full round trip of all event structures.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149564522
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-145978367
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 Application Timeline Server integra...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-90956447
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29863/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-152566468
**[Test build #44685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44685/consoleFull)** for PR 5423 at commit [`3ded4a3`](https://github.com/apache/spark/commit/3ded4a326691f6584bf2aaaddf6d2a8619ba85cd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-107663203
[Test build #33910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33910/consoleFull) for PR 5423 at commit [`9d295bf`](https://github.com/apache/spark/commit/9d295bf9cea733bedd45f0694dce2fed6ae37d1a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28507293
--- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala ---
@@ -0,0 +1,630 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history.yarn
+
+import java.net.{ConnectException, URL}
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger}
+
+import scala.collection.mutable.LinkedList
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.service.{AbstractService, Service}
+import org.apache.hadoop.yarn.api.records.ApplicationId
+import org.apache.hadoop.yarn.api.records.timeline.{TimelineDomain, TimelineEntity, TimelinePutResponse}
+import org.apache.hadoop.yarn.client.api.TimelineClient
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+import org.apache.hadoop.yarn.exceptions.YarnException
+
+import org.apache.spark.deploy.history.yarn.YarnTimelineUtils._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster.YarnService
+import org.apache.spark.{Logging, SparkContext}
+
+/**
+ * Implements a Hadoop service with the init/start logic replaced by that
+ * of the YarnService.
+ * <p>
+ * As <code>AbstractService</code> implements <code>close()</code>, routing
+ * to its <code>stop</code> method, calling <code>close()</code> is sufficient
+ * to stop the service instance.
+ * <p>
+ * However, when registered to receive spark events, the service will continue to
+ * receive them until the spark context is stopped. Events received when this service
+ * is in a <code>STOPPED</code> state will be discarded.
+ */
+private[spark] class YarnHistoryService extends AbstractService("History Service")
+ with YarnService with Logging {
+
+ private var sc: SparkContext = _
+ private var appId: ApplicationId = _
+ private var timelineClient: Option[TimelineClient] = None
+ private var listener: YarnEventListener = _
+ private var appName: String = null
+ private var userName: String = null
+ private var startTime: Long = _
+
+ private var batchSize: Int = YarnHistoryService.DEFAULT_BATCH_SIZE
+
+ // enqueue event to avoid blocking on main thread.
+ private val eventQueue = new LinkedBlockingQueue[TimestampEvent]
+ // cache layer to handle timeline client failure.
+ private var entityList = new LinkedList[TimelineEntity]
+ private var curEntity: Option[TimelineEntity] = None
+ // Do we have enough information filled for the entity
+ private var bAppStart = false
+ private var bAppEnd = false
+ // How many event we saved
+ private var curEventNum = 0
+ private val eventsProcessed: AtomicInteger = new AtomicInteger(0)
+ private val eventsQueued: AtomicInteger = new AtomicInteger(0)
+ private val eventPostFailures: AtomicInteger = new AtomicInteger(0)
+ private val flushCount = new AtomicInteger(0)
+ private var eventHandlingThread: Thread = null
+ private val stopped: AtomicBoolean = new AtomicBoolean(true)
+ private final val lock: AnyRef = new AnyRef
+ private var maxTimeToWaitOnShutdown: Long = YarnHistoryService.SHUTDOWN_WAIT_TIME
+ private var clientFailure = 0
+ private var domainId: String = null
+ private var timelineWebappAddress: URL = _
+
+
+ /**
+ * Create a timeline client
+ * @return the timeline client
+ */
+ def createTimelineClient = {
+ clientFailure += 1
+ logInfo("Creating timelineClient " + clientFailure)
+ val client = TimelineClient.createTimelineClient()
+ client.init(sc.hadoopConfiguration)
+ client.start
+ timelineClient = Some(client)
+ client
+ }
+
+ /**
+ * Get the timeline client; this will create it if needed
+ * @return the client
+ */
+ def getTimelineClient: TimelineClient = {
+ timelineClient.getOrElse(createTimelineClient)
+ }
+
+ def getEventsProcessed: Int = {
+ eventsProcessed.get()
+ }
+
+ def getEventsQueued: Int = {
+ eventsQueued.get()
+ }
+
+ def getQueueSize: Int = {
+ eventQueue.size()
+ }
+
+ def getBatchSize: Int = {
+ batchSize
+ }
+
+ def getEventPostFailures: Int = {
+ eventPostFailures.get()
+ }
+
+ /**
+ * Reset the timeline client
+ * <p>
+ * 1. Stop the timeline client service if running.
+ * 2. set the <code>timelineClient</code> field to <code>None</code>
+ */
+ def resetTimelineClient(): Unit = {
+ stopOptionalService(timelineClient)
+ timelineClient = None
+ }
+
+ /**
+ * Split a comma separated String, filter out any empty items, and return a Set of strings
+ */
+ private def stringToSet(list: String): Set[String] = {
+ list.split(',').map(_.trim).filter(!_.isEmpty).toSet
+ }
+
+ private def createTimelineDomain(): String = {
+ val sparkConf = sc.getConf
+ val aclsOn = sparkConf.getOption("spark.acls.enable").getOrElse(
+ sparkConf.get("spark.ui.acls.enable", "false")).toBoolean
+ if (!aclsOn) {
+ return null
+ }
+ val predefDomain = sparkConf.getOption("spark.ui.domain")
+ if (predefDomain.isDefined) {
+ domainId = predefDomain.get
+ return null
+ }
+ val current = UserGroupInformation.getCurrentUser.getShortUserName
+ val adminAcls = stringToSet(sparkConf.get("spark.admin.acls", ""))
+ val viewAcls = stringToSet(sparkConf.get("spark.ui.view.acls", ""))
+ val modifyAcls = stringToSet(sparkConf.get("spark.modify.acls", ""))
+
+ val readers = (adminAcls ++ modifyAcls ++ viewAcls).foldLeft(current)(_ + " " + _)
+ val writers = (adminAcls ++ modifyAcls).foldLeft(current)(_ + " " + _)
+ var tmpId = YarnHistoryService.DOMAIN_ID_PREFIX + appId
+ logInfo("Creating domain " + tmpId + " with readers: "
+ + readers + " and writers:" + writers)
+ val timelineDomain = new TimelineDomain()
+ timelineDomain.setId(tmpId)
+
+ timelineDomain.setReaders(readers)
+ timelineDomain.setWriters(writers)
+ try {
+ getTimelineClient.putDomain(timelineDomain)
+ } catch {
+ case e: YarnException => {
+ logError("cannot create the domain")
+ // fallback to default
+ tmpId = null
+ }
+ }
+ tmpId
+ }
+
+ def start(context: SparkContext, id: ApplicationId): Boolean = {
+ sc = context
+ appId = id
+ val yarnConf = new YarnConfiguration(context.hadoopConfiguration)
+ // the init() operation checks the state machine & prevents invocation out of sequence
+ init(yarnConf)
+ batchSize = sc.conf.getInt(YarnHistoryService.BATCH_SIZE, batchSize)
+
+ start()
+ if (yarnConf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ true
+ } else {
+ logInfo("Yarn timeline service not available, disabling client.")
+ false
+ }
+ }
+
+ override protected def serviceStart {
+ super.serviceStart()
+ val conf: Configuration = getConfig
+ if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ timelineWebappAddress = timelineWebappUri(conf).toURL
+ createTimelineClient
+ domainId = createTimelineDomain
+ eventHandlingThread = new Thread(new Dequeue(), "HistoryEventHandlingThread")
+ eventHandlingThread.start
+ } else {
+ logInfo("Yarn timeline service not available")
+ }
+ // irrespective of state, hook up to the listener
+ registerListener
+ logInfo(s"$this")
+ }
+
+ /**
+ * Return a summary of the service state to help diagnose problems
+ * during test runs, possibly even production
+ * @return a summary of the current service state
+ */
+ override def toString: String = {
+ super.toString +
+ s" endpoint=$timelineWebappAddress" +
+ s" bonded to ATS=$bondedToATS" +
+ s" listening=$listening" +
+ s" batchSize=$batchSize" +
+ s" flush count=$getFlushCount()" +
+ s" queue size=$getEventsQueued processed=$getEventsProcessed" +
+ s" post failures = $getEventPostFailures"
+ }
+
+ def listening: Boolean = {
+ listener != null;
+ }
+
+ /**
+ * Is the service hooked up to an ATS server. This does not
+ * check the validity of the link, only whether or not the service
+ * has been set up to talk to ATS.
+ * @return true if the service has a timeline client
+ */
+ def bondedToATS: Boolean = {
+ timelineClient != None;
+ }
+
+ /**
+ * Add the listener if it is not disabled.
+ * This is accessible in the same package purely for testing
+ * @return true if the register was enabled
+ */
+ private [yarn] def registerListener: Boolean = {
+ assert(sc != null, "Null context")
+ if (sc.conf.getBoolean(YarnHistoryService.REGISTER_LISTENER, true)) {
+ log.debug("Registering listener to spark context")
+ listener = new YarnEventListener(sc, this)
+ sc.listenerBus.addListener(listener)
+ true
+ } else {
+ log.info("Not listening to spark context events")
+ false
+ }
+ }
+
+ /**
+ * Queue an event, or if the service's <code>stopped</code> flag
+ * is set, discard the event.
+ * @param event
+ * @return true if the event was queued
+ */
+ def enqueue(event: TimestampEvent): Boolean = {
+ if (!stopped.get()) {
+ innerEnqueue(event)
+ true
+ } else {
+ logInfo(s"History service stopped; ignoring queued event : ${event}")
+ false
+ }
+ }
+
+ /**
+ * Inner operation to queue the event. This does not check for service state
+ * @param event
+ */
+ private def innerEnqueue(event: TimestampEvent) = {
+ eventsQueued.incrementAndGet();
+ log.debug(s"Enqueue ${event}")
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-143057274
[Test build #42977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42977/console) for PR 5423 at commit [`9e22fff`](https://github.com/apache/spark/commit/9e22fff9a4bff67f66a391d442d00246a49e9956).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnExtensionService `
* `case class YarnExtensionServiceBinding(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-152537920
**[Test build #44685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44685/consoleFull)** for PR 5423 at commit [`3ded4a3`](https://github.com/apache/spark/commit/3ded4a326691f6584bf2aaaddf6d2a8619ba85cd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162566075
**[Test build #47270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47270/consoleFull)** for PR 5423 at commit [`0adb70a`](https://github.com/apache/spark/commit/0adb70a1240ea4f6fb7f5f78fc54d2facf7ddbef).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147746868
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147823043
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-161405566
**[Test build #47081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47081/consoleFull)** for PR 5423 at commit [`0b89edc`](https://github.com/apache/spark/commit/0b89edcaa00461138861695de7d5085b781d9a7d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28507287
--- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala ---
@@ -0,0 +1,630 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history.yarn
+
+import java.net.{ConnectException, URL}
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger}
+
+import scala.collection.mutable.LinkedList
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.service.{AbstractService, Service}
+import org.apache.hadoop.yarn.api.records.ApplicationId
+import org.apache.hadoop.yarn.api.records.timeline.{TimelineDomain, TimelineEntity, TimelinePutResponse}
+import org.apache.hadoop.yarn.client.api.TimelineClient
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+import org.apache.hadoop.yarn.exceptions.YarnException
+
+import org.apache.spark.deploy.history.yarn.YarnTimelineUtils._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster.YarnService
+import org.apache.spark.{Logging, SparkContext}
+
+/**
+ * Implements a Hadoop service with the init/start logic replaced by that
+ * of the YarnService.
+ * <p>
+ * As <code>AbstractService</code> implements <code>close()</code>, routing
+ * to its <code>stop</code> method, calling <code>close()</code> is sufficient
+ * to stop the service instance.
+ * <p>
+ * However, when registered to receive spark events, the service will continue to
+ * receive them until the spark context is stopped. Events received when this service
+ * is in a <code>STOPPED</code> state will be discarded.
+ */
+private[spark] class YarnHistoryService extends AbstractService("History Service")
+ with YarnService with Logging {
+
+ private var sc: SparkContext = _
+ private var appId: ApplicationId = _
+ private var timelineClient: Option[TimelineClient] = None
+ private var listener: YarnEventListener = _
+ private var appName: String = null
+ private var userName: String = null
+ private var startTime: Long = _
+
+ private var batchSize: Int = YarnHistoryService.DEFAULT_BATCH_SIZE
+
+ // enqueue event to avoid blocking on main thread.
+ private val eventQueue = new LinkedBlockingQueue[TimestampEvent]
+ // cache layer to handle timeline client failure.
+ private var entityList = new LinkedList[TimelineEntity]
+ private var curEntity: Option[TimelineEntity] = None
+ // Do we have enough information filled for the entity
+ private var bAppStart = false
+ private var bAppEnd = false
+ // How many event we saved
+ private var curEventNum = 0
+ private val eventsProcessed: AtomicInteger = new AtomicInteger(0)
+ private val eventsQueued: AtomicInteger = new AtomicInteger(0)
+ private val eventPostFailures: AtomicInteger = new AtomicInteger(0)
+ private val flushCount = new AtomicInteger(0)
+ private var eventHandlingThread: Thread = null
+ private val stopped: AtomicBoolean = new AtomicBoolean(true)
+ private final val lock: AnyRef = new AnyRef
+ private var maxTimeToWaitOnShutdown: Long = YarnHistoryService.SHUTDOWN_WAIT_TIME
+ private var clientFailure = 0
+ private var domainId: String = null
+ private var timelineWebappAddress: URL = _
+
+
+ /**
+ * Create a timeline client
+ * @return the timeline client
+ */
+ def createTimelineClient = {
+ clientFailure += 1
+ logInfo("Creating timelineClient " + clientFailure)
+ val client = TimelineClient.createTimelineClient()
+ client.init(sc.hadoopConfiguration)
+ client.start
+ timelineClient = Some(client)
+ client
+ }
+
+ /**
+ * Get the timeline client; this will create it if needed
+ * @return the client
+ */
+ def getTimelineClient: TimelineClient = {
+ timelineClient.getOrElse(createTimelineClient)
+ }
+
+ def getEventsProcessed: Int = {
+ eventsProcessed.get()
+ }
+
+ def getEventsQueued: Int = {
+ eventsQueued.get()
+ }
+
+ def getQueueSize: Int = {
+ eventQueue.size()
+ }
+
+ def getBatchSize: Int = {
+ batchSize
+ }
+
+ def getEventPostFailures: Int = {
+ eventPostFailures.get()
+ }
+
+ /**
+ * Reset the timeline client
+ * <p>
+ * 1. Stop the timeline client service if running.
+ * 2. set the <code>timelineClient</code> field to <code>None</code>
+ */
+ def resetTimelineClient(): Unit = {
+ stopOptionalService(timelineClient)
+ timelineClient = None
+ }
+
+ /**
+ * Split a comma separated String, filter out any empty items, and return a Set of strings
+ */
+ private def stringToSet(list: String): Set[String] = {
+ list.split(',').map(_.trim).filter(!_.isEmpty).toSet
+ }
+
+ private def createTimelineDomain(): String = {
+ val sparkConf = sc.getConf
+ val aclsOn = sparkConf.getOption("spark.acls.enable").getOrElse(
+ sparkConf.get("spark.ui.acls.enable", "false")).toBoolean
+ if (!aclsOn) {
+ return null
+ }
+ val predefDomain = sparkConf.getOption("spark.ui.domain")
+ if (predefDomain.isDefined) {
+ domainId = predefDomain.get
+ return null
+ }
+ val current = UserGroupInformation.getCurrentUser.getShortUserName
+ val adminAcls = stringToSet(sparkConf.get("spark.admin.acls", ""))
+ val viewAcls = stringToSet(sparkConf.get("spark.ui.view.acls", ""))
+ val modifyAcls = stringToSet(sparkConf.get("spark.modify.acls", ""))
+
+ val readers = (adminAcls ++ modifyAcls ++ viewAcls).foldLeft(current)(_ + " " + _)
+ val writers = (adminAcls ++ modifyAcls).foldLeft(current)(_ + " " + _)
+ var tmpId = YarnHistoryService.DOMAIN_ID_PREFIX + appId
+ logInfo("Creating domain " + tmpId + " with readers: "
+ + readers + " and writers:" + writers)
+ val timelineDomain = new TimelineDomain()
+ timelineDomain.setId(tmpId)
+
+ timelineDomain.setReaders(readers)
+ timelineDomain.setWriters(writers)
+ try {
+ getTimelineClient.putDomain(timelineDomain)
+ } catch {
+ case e: YarnException => {
+ logError("cannot create the domain")
+ // fallback to default
+ tmpId = null
+ }
+ }
+ tmpId
+ }
+
+ def start(context: SparkContext, id: ApplicationId): Boolean = {
+ sc = context
+ appId = id
+ val yarnConf = new YarnConfiguration(context.hadoopConfiguration)
+ // the init() operation checks the state machine & prevents invocation out of sequence
+ init(yarnConf)
+ batchSize = sc.conf.getInt(YarnHistoryService.BATCH_SIZE, batchSize)
+
+ start()
+ if (yarnConf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ true
+ } else {
+ logInfo("Yarn timeline service not available, disabling client.")
+ false
+ }
+ }
+
+ override protected def serviceStart {
+ super.serviceStart()
+ val conf: Configuration = getConfig
+ if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ timelineWebappAddress = timelineWebappUri(conf).toURL
+ createTimelineClient
+ domainId = createTimelineDomain
+ eventHandlingThread = new Thread(new Dequeue(), "HistoryEventHandlingThread")
+ eventHandlingThread.start
+ } else {
+ logInfo("Yarn timeline service not available")
+ }
+ // irrespective of state, hook up to the listener
+ registerListener
+ logInfo(s"$this")
+ }
+
+ /**
+ * Return a summary of the service state to help diagnose problems
+ * during test runs, possibly even production
+ * @return a summary of the current service state
+ */
+ override def toString: String = {
+ super.toString +
+ s" endpoint=$timelineWebappAddress" +
+ s" bonded to ATS=$bondedToATS" +
+ s" listening=$listening" +
+ s" batchSize=$batchSize" +
+ s" flush count=$getFlushCount()" +
+ s" queue size=$getEventsQueued processed=$getEventsProcessed" +
+ s" post failures = $getEventPostFailures"
+ }
+
+ def listening: Boolean = {
+ listener != null;
+ }
+
+ /**
+ * Is the service hooked up to an ATS server. This does not
+ * check the validity of the link, only whether or not the service
+ * has been set up to talk to ATS.
+ * @return true if the service has a timeline client
+ */
+ def bondedToATS: Boolean = {
+ timelineClient != None;
+ }
+
+ /**
+ * Add the listener if it is not disabled.
+ * This is accessible in the same package purely for testing
+ * @return true if the register was enabled
+ */
+ private [yarn] def registerListener: Boolean = {
+ assert(sc != null, "Null context")
+ if (sc.conf.getBoolean(YarnHistoryService.REGISTER_LISTENER, true)) {
+ log.debug("Registering listener to spark context")
+ listener = new YarnEventListener(sc, this)
+ sc.listenerBus.addListener(listener)
+ true
+ } else {
+ log.info("Not listening to spark context events")
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-164084776
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47596/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-91605370
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30029/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28468204
--- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala ---
@@ -0,0 +1,630 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history.yarn
+
+import java.net.{ConnectException, URL}
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger}
+
+import scala.collection.mutable.LinkedList
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.service.{AbstractService, Service}
+import org.apache.hadoop.yarn.api.records.ApplicationId
+import org.apache.hadoop.yarn.api.records.timeline.{TimelineDomain, TimelineEntity, TimelinePutResponse}
+import org.apache.hadoop.yarn.client.api.TimelineClient
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+import org.apache.hadoop.yarn.exceptions.YarnException
+
+import org.apache.spark.deploy.history.yarn.YarnTimelineUtils._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster.YarnService
+import org.apache.spark.{Logging, SparkContext}
+
+/**
+ * Implements a Hadoop service with the init/start logic replaced by that
+ * of the YarnService.
+ * <p>
+ * As <code>AbstractService</code> implements <code>close()</code>, routing
+ * to its <code>stop</code> method, calling <code>close()</code> is sufficient
+ * to stop the service instance.
+ * <p>
+ * However, when registered to receive spark events, the service will continue to
+ * receive them until the spark context is stopped. Events received when this service
+ * is in a <code>STOPPED</code> state will be discarded.
+ */
+private[spark] class YarnHistoryService extends AbstractService("History Service")
+ with YarnService with Logging {
+
+ private var sc: SparkContext = _
+ private var appId: ApplicationId = _
+ private var timelineClient: Option[TimelineClient] = None
+ private var listener: YarnEventListener = _
+ private var appName: String = null
+ private var userName: String = null
+ private var startTime: Long = _
+
+ private var batchSize: Int = YarnHistoryService.DEFAULT_BATCH_SIZE
+
+ // enqueue event to avoid blocking on main thread.
+ private val eventQueue = new LinkedBlockingQueue[TimestampEvent]
+ // cache layer to handle timeline client failure.
+ private var entityList = new LinkedList[TimelineEntity]
+ private var curEntity: Option[TimelineEntity] = None
+ // Do we have enough information filled for the entity
+ private var bAppStart = false
+ private var bAppEnd = false
+ // How many event we saved
+ private var curEventNum = 0
+ private val eventsProcessed: AtomicInteger = new AtomicInteger(0)
+ private val eventsQueued: AtomicInteger = new AtomicInteger(0)
+ private val eventPostFailures: AtomicInteger = new AtomicInteger(0)
+ private val flushCount = new AtomicInteger(0)
+ private var eventHandlingThread: Thread = null
+ private val stopped: AtomicBoolean = new AtomicBoolean(true)
+ private final val lock: AnyRef = new AnyRef
+ private var maxTimeToWaitOnShutdown: Long = YarnHistoryService.SHUTDOWN_WAIT_TIME
+ private var clientFailure = 0
+ private var domainId: String = null
+ private var timelineWebappAddress: URL = _
+
+
+ /**
+ * Create a timeline client
+ * @return the timeline client
+ */
+ def createTimelineClient = {
+ clientFailure += 1
+ logInfo("Creating timelineClient " + clientFailure)
+ val client = TimelineClient.createTimelineClient()
+ client.init(sc.hadoopConfiguration)
+ client.start
+ timelineClient = Some(client)
+ client
+ }
+
+ /**
+ * Get the timeline client; this will create it if needed
+ * @return the client
+ */
+ def getTimelineClient: TimelineClient = {
+ timelineClient.getOrElse(createTimelineClient)
+ }
+
+ def getEventsProcessed: Int = {
+ eventsProcessed.get()
+ }
+
+ def getEventsQueued: Int = {
+ eventsQueued.get()
+ }
+
+ def getQueueSize: Int = {
+ eventQueue.size()
+ }
+
+ def getBatchSize: Int = {
+ batchSize
+ }
+
+ def getEventPostFailures: Int = {
+ eventPostFailures.get()
+ }
+
+ /**
+ * Reset the timeline client
+ * <p>
+ * 1. Stop the timeline client service if running.
+ * 2. set the <code>timelineClient</code> field to <code>None</code>
+ */
+ def resetTimelineClient(): Unit = {
+ stopOptionalService(timelineClient)
+ timelineClient = None
+ }
+
+ /**
+ * Split a comma separated String, filter out any empty items, and return a Set of strings
+ */
+ private def stringToSet(list: String): Set[String] = {
+ list.split(',').map(_.trim).filter(!_.isEmpty).toSet
+ }
+
+ private def createTimelineDomain(): String = {
+ val sparkConf = sc.getConf
+ val aclsOn = sparkConf.getOption("spark.acls.enable").getOrElse(
+ sparkConf.get("spark.ui.acls.enable", "false")).toBoolean
+ if (!aclsOn) {
+ return null
+ }
+ val predefDomain = sparkConf.getOption("spark.ui.domain")
+ if (predefDomain.isDefined) {
+ domainId = predefDomain.get
+ return null
+ }
+ val current = UserGroupInformation.getCurrentUser.getShortUserName
+ val adminAcls = stringToSet(sparkConf.get("spark.admin.acls", ""))
+ val viewAcls = stringToSet(sparkConf.get("spark.ui.view.acls", ""))
+ val modifyAcls = stringToSet(sparkConf.get("spark.modify.acls", ""))
+
+ val readers = (adminAcls ++ modifyAcls ++ viewAcls).foldLeft(current)(_ + " " + _)
+ val writers = (adminAcls ++ modifyAcls).foldLeft(current)(_ + " " + _)
+ var tmpId = YarnHistoryService.DOMAIN_ID_PREFIX + appId
+ logInfo("Creating domain " + tmpId + " with readers: "
+ + readers + " and writers:" + writers)
+ val timelineDomain = new TimelineDomain()
+ timelineDomain.setId(tmpId)
+
+ timelineDomain.setReaders(readers)
+ timelineDomain.setWriters(writers)
+ try {
+ getTimelineClient.putDomain(timelineDomain)
+ } catch {
+ case e: YarnException => {
+ logError("cannot create the domain")
+ // fallback to default
+ tmpId = null
+ }
+ }
+ tmpId
+ }
+
+ def start(context: SparkContext, id: ApplicationId): Boolean = {
+ sc = context
+ appId = id
+ val yarnConf = new YarnConfiguration(context.hadoopConfiguration)
+ // the init() operation checks the state machine & prevents invocation out of sequence
+ init(yarnConf)
+ batchSize = sc.conf.getInt(YarnHistoryService.BATCH_SIZE, batchSize)
+
+ start()
+ if (yarnConf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ true
+ } else {
+ logInfo("Yarn timeline service not available, disabling client.")
+ false
+ }
+ }
+
+ override protected def serviceStart {
+ super.serviceStart()
+ val conf: Configuration = getConfig
+ if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ timelineWebappAddress = timelineWebappUri(conf).toURL
+ createTimelineClient
+ domainId = createTimelineDomain
+ eventHandlingThread = new Thread(new Dequeue(), "HistoryEventHandlingThread")
+ eventHandlingThread.start
+ } else {
+ logInfo("Yarn timeline service not available")
+ }
+ // irrespective of state, hook up to the listener
+ registerListener
+ logInfo(s"$this")
+ }
+
+ /**
+ * Return a summary of the service state to help diagnose problems
+ * during test runs, possibly even production
+ * @return a summary of the current service state
+ */
+ override def toString: String = {
+ super.toString +
+ s" endpoint=$timelineWebappAddress" +
+ s" bonded to ATS=$bondedToATS" +
+ s" listening=$listening" +
+ s" batchSize=$batchSize" +
+ s" flush count=$getFlushCount()" +
+ s" queue size=$getEventsQueued processed=$getEventsProcessed" +
+ s" post failures = $getEventPostFailures"
+ }
+
+ def listening: Boolean = {
+ listener != null;
+ }
+
+ /**
+ * Is the service hooked up to an ATS server. This does not
+ * check the validity of the link, only whether or not the service
+ * has been set up to talk to ATS.
+ * @return true if the service has a timeline client
+ */
+ def bondedToATS: Boolean = {
+ timelineClient != None;
+ }
+
+ /**
+ * Add the listener if it is not disabled.
+ * This is accessible in the same package purely for testing
+ * @return true if the register was enabled
+ */
+ private [yarn] def registerListener: Boolean = {
+ assert(sc != null, "Null context")
+ if (sc.conf.getBoolean(YarnHistoryService.REGISTER_LISTENER, true)) {
+ log.debug("Registering listener to spark context")
--- End diff --
use logDebug
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-163644585
**[Test build #47495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47495/consoleFull)** for PR 5423 at commit [`52b9a2e`](https://github.com/apache/spark/commit/52b9a2e9d918fcc2965f5db606c44957e7d230f4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-156398548
**[Test build #45859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45859/consoleFull)** for PR 5423 at commit [`ccfeba7`](https://github.com/apache/spark/commit/ccfeba704611a78cf27f77f78abfae420c45da42).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-107693692
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-107693675
[Test build #33910 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33910/consoleFull) for PR 5423 at commit [`9d295bf`](https://github.com/apache/spark/commit/9d295bf9cea733bedd45f0694dce2fed6ae37d1a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnExtensionService extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95707675
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30862/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139044610
[Test build #42212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42212/console) for PR 5423 at commit [`1736bb6`](https://github.com/apache/spark/commit/1736bb68cb5b0676d38c208a4255924618682687).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnExtensionService extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-97196825
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147783276
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139374475
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139331889
[Test build #42278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42278/consoleFull) for PR 5423 at commit [`5c259d2`](https://github.com/apache/spark/commit/5c259d26ca98c1645ab31f9597df2f237361a90c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162997897
**[Test build #47346 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47346/consoleFull)** for PR 5423 at commit [`b0e25bd`](https://github.com/apache/spark/commit/b0e25bdbcf16456791dd908fb2508dc0a22ad9db).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-146022350
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-149568100
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-168318509
now succeeded by #10545
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-143314734
> Move history to src to keep consistent like other modules.
I like that. `yarn/src/history/scala` looks cleaner than the current approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139001166
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-164084728
**[Test build #47596 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47596/consoleFull)** for PR 5423 at commit [`e134e29`](https://github.com/apache/spark/commit/e134e29c8ae121dbacb9e09c70e0c4820915c296).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139044747
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42212/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139374476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42278/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147746919
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95314470
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30770/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-152566556
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 Application Timeline Server integra...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-90956432
[Test build #29863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29863/consoleFull) for PR 5423 at commit [`1543f47`](https://github.com/apache/spark/commit/1543f47ad06c218f818cd4fa6bb88f2fea91607d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class TimestampEvent(sparkEvent: SparkListenerEvent, time: Long) `
* `class YarnEventListener(sc: SparkContext, service: YarnHistoryService)`
* `class YarnHistoryProvider(sparkConf: SparkConf)`
* `class YarnHistoryService extends AbstractService("History Service")`
* `class WeakShutdownHook(service: YarnService) extends Runnable with Logging `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139002690
[Test build #42213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42213/consoleFull) for PR 5423 at commit [`c16f112`](https://github.com/apache/spark/commit/c16f1129336c5383c8e0bd62dcdb3f6425a5bfde).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95675032
There's no obvious reason why the Jenkins build failed; the console says all the tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151974958
**[Test build #44539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44539/consoleFull)** for PR 5423 at commit [`6057521`](https://github.com/apache/spark/commit/6057521fd1e14d7ac5eb60fd5650b37c0d2dd2d5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139001193
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151934300
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151825740
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95314452
[Test build #30770 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30770/consoleFull) for PR 5423 at commit [`f8509b8`](https://github.com/apache/spark/commit/f8509b8d0cb6ab274bb1c1a1cba576da29a8c546).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnService extends Closeable `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139044744
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-143022238
[Test build #42977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42977/consoleFull) for PR 5423 at commit [`9e22fff`](https://github.com/apache/spark/commit/9e22fff9a4bff67f66a391d442d00246a49e9956).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-140048781
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-162998152
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47346/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151875771
**[Test build #44526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44526/consoleFull)** for PR 5423 at commit [`2c1db93`](https://github.com/apache/spark/commit/2c1db93bb1fe72a03e4b866741b6b803b30bb2b3).
* This patch passes all tests.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95281268
[Test build #30770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30770/consoleFull) for PR 5423 at commit [`f8509b8`](https://github.com/apache/spark/commit/f8509b8d0cb6ab274bb1c1a1cba576da29a8c546).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-163644629
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94949294
[Test build #30695 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30695/consoleFull) for PR 5423 at commit [`4c0dd85`](https://github.com/apache/spark/commit/4c0dd85d0e3916c10f756fcc33e075924eddce58).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait YarnService extends Closeable `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94593519
[Test build #30605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30605/consoleFull) for PR 5423 at commit [`0f0f66c`](https://github.com/apache/spark/commit/0f0f66c810a0ea3c647ddf70e73c2a79e932d2f7).
* This patch **passes all tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `case class TimestampEvent(sparkEvent: SparkListenerEvent, time: Long, flush: Boolean = false) `
* `class YarnHistoryProvider(sparkConf: SparkConf)`
* `trait YarnService extends Closeable `
* This patch **adds the following new dependencies:**
* `commons-math3-3.1.1.jar`
* `snappy-java-1.1.1.6.jar`
* This patch **removes the following dependencies:**
* `commons-math3-3.4.1.jar`
* `snappy-java-1.1.1.7.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-94830807
Note that this adds a new profile `hadoop-2.6`, to pull in the 2.6 JARs and conditionally add yarn/history source & tests to the build...without that the tests (probably) aren't running. I say probably as whatever is looking at public interfaces/classes does appear to be looking into history/src/main/scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-145978386
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-143057360
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28468238
--- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala ---
@@ -0,0 +1,630 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history.yarn
+
+import java.net.{ConnectException, URL}
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger}
+
+import scala.collection.mutable.LinkedList
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.service.{AbstractService, Service}
+import org.apache.hadoop.yarn.api.records.ApplicationId
+import org.apache.hadoop.yarn.api.records.timeline.{TimelineDomain, TimelineEntity, TimelinePutResponse}
+import org.apache.hadoop.yarn.client.api.TimelineClient
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+import org.apache.hadoop.yarn.exceptions.YarnException
+
+import org.apache.spark.deploy.history.yarn.YarnTimelineUtils._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster.YarnService
+import org.apache.spark.{Logging, SparkContext}
+
+/**
+ * Implements a Hadoop service with the init/start logic replaced by that
+ * of the YarnService.
+ * <p>
+ * As <code>AbstractService</code> implements <code>close()</code>, routing
+ * to its <code>stop</code> method, calling <code>close()</code> is sufficient
+ * to stop the service instance.
+ * <p>
+ * However, when registered to receive spark events, the service will continue to
+ * receive them until the spark context is stopped. Events received when this service
+ * is in a <code>STOPPED</code> state will be discarded.
+ */
+private[spark] class YarnHistoryService extends AbstractService("History Service")
+ with YarnService with Logging {
+
+ private var sc: SparkContext = _
+ private var appId: ApplicationId = _
+ private var timelineClient: Option[TimelineClient] = None
+ private var listener: YarnEventListener = _
+ private var appName: String = null
+ private var userName: String = null
+ private var startTime: Long = _
+
+ private var batchSize: Int = YarnHistoryService.DEFAULT_BATCH_SIZE
+
+ // enqueue event to avoid blocking on main thread.
+ private val eventQueue = new LinkedBlockingQueue[TimestampEvent]
+ // cache layer to handle timeline client failure.
+ private var entityList = new LinkedList[TimelineEntity]
+ private var curEntity: Option[TimelineEntity] = None
+ // Do we have enough information filled for the entity
+ private var bAppStart = false
+ private var bAppEnd = false
+ // How many event we saved
+ private var curEventNum = 0
+ private val eventsProcessed: AtomicInteger = new AtomicInteger(0)
+ private val eventsQueued: AtomicInteger = new AtomicInteger(0)
+ private val eventPostFailures: AtomicInteger = new AtomicInteger(0)
+ private val flushCount = new AtomicInteger(0)
+ private var eventHandlingThread: Thread = null
+ private val stopped: AtomicBoolean = new AtomicBoolean(true)
+ private final val lock: AnyRef = new AnyRef
+ private var maxTimeToWaitOnShutdown: Long = YarnHistoryService.SHUTDOWN_WAIT_TIME
+ private var clientFailure = 0
+ private var domainId: String = null
+ private var timelineWebappAddress: URL = _
+
+
+ /**
+ * Create a timeline client
+ * @return the timeline client
+ */
+ def createTimelineClient = {
+ clientFailure += 1
+ logInfo("Creating timelineClient " + clientFailure)
+ val client = TimelineClient.createTimelineClient()
+ client.init(sc.hadoopConfiguration)
+ client.start
+ timelineClient = Some(client)
+ client
+ }
+
+ /**
+ * Get the timeline client; this will create it if needed
+ * @return the client
+ */
+ def getTimelineClient: TimelineClient = {
+ timelineClient.getOrElse(createTimelineClient)
+ }
+
+ def getEventsProcessed: Int = {
+ eventsProcessed.get()
+ }
+
+ def getEventsQueued: Int = {
+ eventsQueued.get()
+ }
+
+ def getQueueSize: Int = {
+ eventQueue.size()
+ }
+
+ def getBatchSize: Int = {
+ batchSize
+ }
+
+ def getEventPostFailures: Int = {
+ eventPostFailures.get()
+ }
+
+ /**
+ * Reset the timeline client
+ * <p>
+ * 1. Stop the timeline client service if running.
+ * 2. set the <code>timelineClient</code> field to <code>None</code>
+ */
+ def resetTimelineClient(): Unit = {
+ stopOptionalService(timelineClient)
+ timelineClient = None
+ }
+
+ /**
+ * Split a comma separated String, filter out any empty items, and return a Set of strings
+ */
+ private def stringToSet(list: String): Set[String] = {
+ list.split(',').map(_.trim).filter(!_.isEmpty).toSet
+ }
+
+ private def createTimelineDomain(): String = {
+ val sparkConf = sc.getConf
+ val aclsOn = sparkConf.getOption("spark.acls.enable").getOrElse(
+ sparkConf.get("spark.ui.acls.enable", "false")).toBoolean
+ if (!aclsOn) {
+ return null
+ }
+ val predefDomain = sparkConf.getOption("spark.ui.domain")
+ if (predefDomain.isDefined) {
+ domainId = predefDomain.get
+ return null
+ }
+ val current = UserGroupInformation.getCurrentUser.getShortUserName
+ val adminAcls = stringToSet(sparkConf.get("spark.admin.acls", ""))
+ val viewAcls = stringToSet(sparkConf.get("spark.ui.view.acls", ""))
+ val modifyAcls = stringToSet(sparkConf.get("spark.modify.acls", ""))
+
+ val readers = (adminAcls ++ modifyAcls ++ viewAcls).foldLeft(current)(_ + " " + _)
+ val writers = (adminAcls ++ modifyAcls).foldLeft(current)(_ + " " + _)
+ var tmpId = YarnHistoryService.DOMAIN_ID_PREFIX + appId
+ logInfo("Creating domain " + tmpId + " with readers: "
+ + readers + " and writers:" + writers)
+ val timelineDomain = new TimelineDomain()
+ timelineDomain.setId(tmpId)
+
+ timelineDomain.setReaders(readers)
+ timelineDomain.setWriters(writers)
+ try {
+ getTimelineClient.putDomain(timelineDomain)
+ } catch {
+ case e: YarnException => {
+ logError("cannot create the domain")
+ // fallback to default
+ tmpId = null
+ }
+ }
+ tmpId
+ }
+
+ def start(context: SparkContext, id: ApplicationId): Boolean = {
+ sc = context
+ appId = id
+ val yarnConf = new YarnConfiguration(context.hadoopConfiguration)
+ // the init() operation checks the state machine & prevents invocation out of sequence
+ init(yarnConf)
+ batchSize = sc.conf.getInt(YarnHistoryService.BATCH_SIZE, batchSize)
+
+ start()
+ if (yarnConf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ true
+ } else {
+ logInfo("Yarn timeline service not available, disabling client.")
+ false
+ }
+ }
+
+ override protected def serviceStart {
+ super.serviceStart()
+ val conf: Configuration = getConfig
+ if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ timelineWebappAddress = timelineWebappUri(conf).toURL
+ createTimelineClient
+ domainId = createTimelineDomain
+ eventHandlingThread = new Thread(new Dequeue(), "HistoryEventHandlingThread")
+ eventHandlingThread.start
+ } else {
+ logInfo("Yarn timeline service not available")
+ }
+ // irrespective of state, hook up to the listener
+ registerListener
+ logInfo(s"$this")
+ }
+
+ /**
+ * Return a summary of the service state to help diagnose problems
+ * during test runs, possibly even production
+ * @return a summary of the current service state
+ */
+ override def toString: String = {
+ super.toString +
+ s" endpoint=$timelineWebappAddress" +
+ s" bonded to ATS=$bondedToATS" +
+ s" listening=$listening" +
+ s" batchSize=$batchSize" +
+ s" flush count=$getFlushCount()" +
+ s" queue size=$getEventsQueued processed=$getEventsProcessed" +
+ s" post failures = $getEventPostFailures"
+ }
+
+ def listening: Boolean = {
+ listener != null;
+ }
+
+ /**
+ * Is the service hooked up to an ATS server. This does not
+ * check the validity of the link, only whether or not the service
+ * has been set up to talk to ATS.
+ * @return true if the service has a timeline client
+ */
+ def bondedToATS: Boolean = {
+ timelineClient != None;
+ }
+
+ /**
+ * Add the listener if it is not disabled.
+ * This is accessible in the same package purely for testing
+ * @return true if the register was enabled
+ */
+ private [yarn] def registerListener: Boolean = {
+ assert(sc != null, "Null context")
+ if (sc.conf.getBoolean(YarnHistoryService.REGISTER_LISTENER, true)) {
+ log.debug("Registering listener to spark context")
+ listener = new YarnEventListener(sc, this)
+ sc.listenerBus.addListener(listener)
+ true
+ } else {
+ log.info("Not listening to spark context events")
+ false
+ }
+ }
+
+ /**
+ * Queue an event, or if the service's <code>stopped</code> flag
+ * is set, discard the event.
+ * @param event
+ * @return true if the event was queued
+ */
+ def enqueue(event: TimestampEvent): Boolean = {
+ if (!stopped.get()) {
+ innerEnqueue(event)
+ true
+ } else {
+ logInfo(s"History service stopped; ignoring queued event : ${event}")
+ false
+ }
+ }
+
+ /**
+ * Inner operation to queue the event. This does not check for service state
+ * @param event
+ */
+ private def innerEnqueue(event: TimestampEvent) = {
+ eventsQueued.incrementAndGet();
+ log.debug(s"Enqueue ${event}")
--- End diff --
logDebug
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-97243948
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31164/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95350799
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30781/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151260665
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44370/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-147698799
I've been thinking about this —how about I do the following
# have move the history in under yarn/src/history, as discussed above
# also as discussed isolate the patch to the scheduler extension services —this includes converting some tests to be isolated, while moving some the Stub* implementations of the YARN applicationId. attemptId &c into the main {{yarn/src/test}} source tree. That way this can be reviewed & committed independently of the others.
This is all relatively straightforward -if people are happy I can have it done within 24h
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-95305828
[Test build #30781 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30781/consoleFull) for PR 5423 at commit [`957daf5`](https://github.com/apache/spark/commit/957daf5ccfade15cd45bf0cc56f4a02c43f1f3e2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-139004547
I've not been pushing this up much. Current status
1. multiple attempts supported: every event -> a separate entity; these get aggregated for the reports.
1. more efficient listing of apps/attempts means that background refreshes put less load on ATS, so the scheduled background refresh code is re-instated...though the on-demand option is still there and can trigger faster updates.
1. The RM gets queried for running spark apps —the records are used to recognise attempts that finished without a record, e.g. AM crash. The policy is
* failed/finished/killed ==> completed
* Not in the list and the ApplicationAttemptInfo last updated over an hour ago: assume failed. (The hour long delay is to handle transient loss of apps from listing)
Hitting the RM for status on running apps makes a big difference; it keeps killed apps out the incomplete list.
This is what the view looks like with some single attempt apps & double attempts.
I plan to to go from listing attempts by yarn attempt ID to the simpler "1", "2", "3" numbering; most of the code to do that is there, I've just not switched over to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28507280
--- Diff: yarn/history/src/main/scala/org/apache/spark/deploy/history/yarn/YarnHistoryService.scala ---
@@ -0,0 +1,630 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history.yarn
+
+import java.net.{ConnectException, URL}
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger}
+
+import scala.collection.mutable.LinkedList
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.hadoop.service.{AbstractService, Service}
+import org.apache.hadoop.yarn.api.records.ApplicationId
+import org.apache.hadoop.yarn.api.records.timeline.{TimelineDomain, TimelineEntity, TimelinePutResponse}
+import org.apache.hadoop.yarn.client.api.TimelineClient
+import org.apache.hadoop.yarn.conf.YarnConfiguration
+import org.apache.hadoop.yarn.exceptions.YarnException
+
+import org.apache.spark.deploy.history.yarn.YarnTimelineUtils._
+import org.apache.spark.scheduler._
+import org.apache.spark.scheduler.cluster.YarnService
+import org.apache.spark.{Logging, SparkContext}
+
+/**
+ * Implements a Hadoop service with the init/start logic replaced by that
+ * of the YarnService.
+ * <p>
+ * As <code>AbstractService</code> implements <code>close()</code>, routing
+ * to its <code>stop</code> method, calling <code>close()</code> is sufficient
+ * to stop the service instance.
+ * <p>
+ * However, when registered to receive spark events, the service will continue to
+ * receive them until the spark context is stopped. Events received when this service
+ * is in a <code>STOPPED</code> state will be discarded.
+ */
+private[spark] class YarnHistoryService extends AbstractService("History Service")
+ with YarnService with Logging {
+
+ private var sc: SparkContext = _
+ private var appId: ApplicationId = _
+ private var timelineClient: Option[TimelineClient] = None
+ private var listener: YarnEventListener = _
+ private var appName: String = null
+ private var userName: String = null
+ private var startTime: Long = _
+
+ private var batchSize: Int = YarnHistoryService.DEFAULT_BATCH_SIZE
+
+ // enqueue event to avoid blocking on main thread.
+ private val eventQueue = new LinkedBlockingQueue[TimestampEvent]
+ // cache layer to handle timeline client failure.
+ private var entityList = new LinkedList[TimelineEntity]
+ private var curEntity: Option[TimelineEntity] = None
+ // Do we have enough information filled for the entity
+ private var bAppStart = false
+ private var bAppEnd = false
+ // How many event we saved
+ private var curEventNum = 0
+ private val eventsProcessed: AtomicInteger = new AtomicInteger(0)
+ private val eventsQueued: AtomicInteger = new AtomicInteger(0)
+ private val eventPostFailures: AtomicInteger = new AtomicInteger(0)
+ private val flushCount = new AtomicInteger(0)
+ private var eventHandlingThread: Thread = null
+ private val stopped: AtomicBoolean = new AtomicBoolean(true)
+ private final val lock: AnyRef = new AnyRef
+ private var maxTimeToWaitOnShutdown: Long = YarnHistoryService.SHUTDOWN_WAIT_TIME
+ private var clientFailure = 0
+ private var domainId: String = null
+ private var timelineWebappAddress: URL = _
+
+
+ /**
+ * Create a timeline client
+ * @return the timeline client
+ */
+ def createTimelineClient = {
+ clientFailure += 1
+ logInfo("Creating timelineClient " + clientFailure)
+ val client = TimelineClient.createTimelineClient()
+ client.init(sc.hadoopConfiguration)
+ client.start
+ timelineClient = Some(client)
+ client
+ }
+
+ /**
+ * Get the timeline client; this will create it if needed
+ * @return the client
+ */
+ def getTimelineClient: TimelineClient = {
+ timelineClient.getOrElse(createTimelineClient)
+ }
+
+ def getEventsProcessed: Int = {
+ eventsProcessed.get()
+ }
+
+ def getEventsQueued: Int = {
+ eventsQueued.get()
+ }
+
+ def getQueueSize: Int = {
+ eventQueue.size()
+ }
+
+ def getBatchSize: Int = {
+ batchSize
+ }
+
+ def getEventPostFailures: Int = {
+ eventPostFailures.get()
+ }
+
+ /**
+ * Reset the timeline client
+ * <p>
+ * 1. Stop the timeline client service if running.
+ * 2. set the <code>timelineClient</code> field to <code>None</code>
+ */
+ def resetTimelineClient(): Unit = {
+ stopOptionalService(timelineClient)
+ timelineClient = None
+ }
+
+ /**
+ * Split a comma separated String, filter out any empty items, and return a Set of strings
+ */
+ private def stringToSet(list: String): Set[String] = {
+ list.split(',').map(_.trim).filter(!_.isEmpty).toSet
+ }
+
+ private def createTimelineDomain(): String = {
+ val sparkConf = sc.getConf
+ val aclsOn = sparkConf.getOption("spark.acls.enable").getOrElse(
+ sparkConf.get("spark.ui.acls.enable", "false")).toBoolean
+ if (!aclsOn) {
+ return null
+ }
+ val predefDomain = sparkConf.getOption("spark.ui.domain")
+ if (predefDomain.isDefined) {
+ domainId = predefDomain.get
+ return null
+ }
+ val current = UserGroupInformation.getCurrentUser.getShortUserName
+ val adminAcls = stringToSet(sparkConf.get("spark.admin.acls", ""))
+ val viewAcls = stringToSet(sparkConf.get("spark.ui.view.acls", ""))
+ val modifyAcls = stringToSet(sparkConf.get("spark.modify.acls", ""))
+
+ val readers = (adminAcls ++ modifyAcls ++ viewAcls).foldLeft(current)(_ + " " + _)
+ val writers = (adminAcls ++ modifyAcls).foldLeft(current)(_ + " " + _)
+ var tmpId = YarnHistoryService.DOMAIN_ID_PREFIX + appId
+ logInfo("Creating domain " + tmpId + " with readers: "
+ + readers + " and writers:" + writers)
+ val timelineDomain = new TimelineDomain()
+ timelineDomain.setId(tmpId)
+
+ timelineDomain.setReaders(readers)
+ timelineDomain.setWriters(writers)
+ try {
+ getTimelineClient.putDomain(timelineDomain)
+ } catch {
+ case e: YarnException => {
+ logError("cannot create the domain")
+ // fallback to default
+ tmpId = null
+ }
+ }
+ tmpId
+ }
+
+ def start(context: SparkContext, id: ApplicationId): Boolean = {
+ sc = context
+ appId = id
+ val yarnConf = new YarnConfiguration(context.hadoopConfiguration)
+ // the init() operation checks the state machine & prevents invocation out of sequence
+ init(yarnConf)
+ batchSize = sc.conf.getInt(YarnHistoryService.BATCH_SIZE, batchSize)
+
+ start()
+ if (yarnConf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ true
+ } else {
+ logInfo("Yarn timeline service not available, disabling client.")
+ false
+ }
+ }
+
+ override protected def serviceStart {
+ super.serviceStart()
+ val conf: Configuration = getConfig
+ if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
+ timelineWebappAddress = timelineWebappUri(conf).toURL
+ createTimelineClient
+ domainId = createTimelineDomain
+ eventHandlingThread = new Thread(new Dequeue(), "HistoryEventHandlingThread")
+ eventHandlingThread.start
+ } else {
+ logInfo("Yarn timeline service not available")
+ }
+ // irrespective of state, hook up to the listener
+ registerListener
+ logInfo(s"$this")
+ }
+
+ /**
+ * Return a summary of the service state to help diagnose problems
+ * during test runs, possibly even production
+ * @return a summary of the current service state
+ */
+ override def toString: String = {
+ super.toString +
+ s" endpoint=$timelineWebappAddress" +
+ s" bonded to ATS=$bondedToATS" +
+ s" listening=$listening" +
+ s" batchSize=$batchSize" +
+ s" flush count=$getFlushCount()" +
+ s" queue size=$getEventsQueued processed=$getEventsProcessed" +
+ s" post failures = $getEventPostFailures"
+ }
+
+ def listening: Boolean = {
+ listener != null;
+ }
+
+ /**
+ * Is the service hooked up to an ATS server. This does not
+ * check the validity of the link, only whether or not the service
+ * has been set up to talk to ATS.
+ * @return true if the service has a timeline client
+ */
+ def bondedToATS: Boolean = {
+ timelineClient != None;
+ }
+
+ /**
+ * Add the listener if it is not disabled.
+ * This is accessible in the same package purely for testing
+ * @return true if the register was enabled
+ */
+ private [yarn] def registerListener: Boolean = {
+ assert(sc != null, "Null context")
+ if (sc.conf.getBoolean(YarnHistoryService.REGISTER_LISTENER, true)) {
+ log.debug("Registering listener to spark context")
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-151220020
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/5423#discussion_r28169146
--- Diff: yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala ---
@@ -56,10 +59,16 @@ private[spark] class YarnClientSchedulerBackend(
totalExpectedExecutors = args.numExecutors
client = new Client(args, conf)
appId = client.submitApplication()
+ services.start(sc, appId)
+ services.addShutdownHook(ApplicationMaster.SHUTDOWN_HOOK_PRIORITY - 1)
--- End diff --
Do you still need to do the same thing in YarnClusterSchedulerBackend? or spark-1.4 consolidate then together?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 Application Timeline Server integra...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-90925113
[Test build #29863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29863/consoleFull) for PR 5423 at commit [`1543f47`](https://github.com/apache/spark/commit/1543f47ad06c218f818cd4fa6bb88f2fea91607d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1537 [WiP] Application Timeline Server i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-138996223
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1537] [YARN] [WiP] Add history provider...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/5423#issuecomment-168315481
yes, it is still relevant, yes it was awaiting review, no I wasn't expecting it to be closed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org