You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sarutak <gi...@git.apache.org> on 2014/09/09 12:08:59 UTC
[GitHub] spark pull request: Implemented renderJson for StagePage
GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/2333
Implemented renderJson for StagePage
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark json-exposing-feature
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2333.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2333
----
commit 1cbf7d5e3583c41fc2bd20c77b73f204587b15f7
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2014-09-09T10:08:27Z
Implemented renderJson for StagePage
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55316029
Hi @sarutak, thanks for working on this feature. There has been request of this on the mailing list and it's good to see this being done. There are a lot of style issues with the current state of the PR, however. Another thing is I don't think we need to prepend the category to the Json blob (e.g. "Environment" or "RDDs"), since the user should know what they asked for.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55554685
test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55513182
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20289/consoleFull) for PR 2333 at commit [`7f51a4f`](https://github.com/apache/spark/commit/7f51a4f1a979221ba8685662a66170c42f0447e9).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55357150
@andrewor14 Thank you for reviewing this PR.
I agree with removing category from the JSON blob and have done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55966735
test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55554859
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20326/consoleFull) for PR 2333 at commit [`d41b3ca`](https://github.com/apache/spark/commit/d41b3caf1adb0c807aa6ce9d011e5e2553408fe2).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443083
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala ---
@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
private[ui] class EnvironmentPage(parent: EnvironmentTab) extends WebUIPage("") {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+ val jvmInfoJson =
+ ("RUntime Informationf" -> listener.jvmInformation.foldLeft(JObject())(_ ~ _))
--- End diff --
Can you show what the final JSON looks like? Since this is a map it might be good to convert this faithfully, e.g.
```
{
"Spark Properties": {
"spark.master":"local"
"spark.app.name":"spark-shell"
...
}
...
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443185
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala ---
@@ -44,6 +47,30 @@ private case class ExecutorSummaryInfo(
private[ui] class ExecutorsPage(parent: ExecutorsTab) extends WebUIPage("") {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+ val storageStatusList = listener.storageStatusList
+
+ val execInfoJsonList = for (statusId <- 0 until storageStatusList.size) yield {
+ val execInfo = getExecInfo(statusId)
+ ("Executor ID" -> execInfo.id) ~
+ ("Address" -> execInfo.hostPort) ~
+ ("RDD Blocks" -> execInfo.rddBlocks) ~
+ ("Memory Used" -> execInfo.memoryUsed) ~
+ ("Disk Used" -> execInfo.diskUsed) ~
+ ("Active Tasks" -> execInfo.activeTasks) ~
+ ("Failed Tasks" -> execInfo.failedTasks) ~
+ ("Complete Tasks" -> execInfo.completedTasks) ~
+ ("TotalTasks" -> execInfo.totalTasks) ~
--- End diff --
space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-56419802
I've opened [SPARK-3644](https://issues.apache.org/jira/browse/SPARK-3644) as a forum for discussing the design of a REST API; sorry for the delay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: (WIP) [SPARK-3454] Expose JSON representation ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55231399
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20143/consoleFull) for PR 2333 at commit [`8850706`](https://github.com/apache/spark/commit/88507061c3c491af47831162a14763d685df3b83).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443289
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/PoolPage.scala ---
@@ -30,6 +35,35 @@ private[ui] class PoolPage(parent: JobProgressTab) extends WebUIPage("pool") {
private val sc = parent.sc
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+ listener.synchronized {
+ val poolName = request.getParameter("poolname")
+ val poolToActiveStages = listener.poolToActiveStages
+ val activeStages = poolToActiveStages.get(poolName) match {
+ case Some(s) => s.values.map {
+ case info: StageInfo =>
+ JsonProtocol.stageInfoToJson(info)
+ }
+ case None => Seq[JValue]()
+ }
+
+ val pools = if (live) Seq(sc.getPoolForName(poolName).get) else Seq[Schedulable]()
+
+ val poolList = pools.map {
+ case schedulable: Schedulable =>
--- End diff --
same here, no need to do `case` and specify the type
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443244
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -31,6 +36,35 @@ private[ui] class JobProgressPage(parent: JobProgressTab) extends WebUIPage("")
private val listener = parent.listener
private lazy val isFairScheduler = parent.isFairScheduler
+ override def renderJson(request: HttpServletRequest): JValue = {
+ listener.synchronized {
+
+ val activeStageList = listener.activeStages.values.map {
+ case info: StageInfo =>
+ JsonProtocol.stageInfoToJson(info)
+ }
--- End diff --
here and other places
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55359688
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20200/consoleFull) for PR 2333 at commit [`72c0644`](https://github.com/apache/spark/commit/72c06445211579a207b36de2d4072974fe7325de).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55978533
Thanks @JoshRosen !
So, for now, I use self implementation for #2342 and I'll use the feature you'll try.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: (WIP) [SPARK-3454] Expose JSON representation ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-54953090
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20035/consoleFull) for PR 2333 at commit [`1cbf7d5`](https://github.com/apache/spark/commit/1cbf7d5e3583c41fc2bd20c77b73f204587b15f7).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-71532457
Hi @sarutak thanks for your work on this. Josh's other PR https://github.com/apache/spark/pull/2696 has been merged for a while now. I'm gonna take another crack at implementing this on top of the lastest changes and getting this integrated into the history server too. Can you please close this PR (unless you are planning on updating it in the very near future)?
Hopefully that will let your other PR https://github.com/apache/spark/pull/2342 to get merged in -- looks really cool!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-56677764
Thank you for you work @JoshRosen !
I'll check it out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak closed the pull request at:
https://github.com/apache/spark/pull/2333
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55976627
@andrewor14 Thank you for notification!
Actually, I need JSON representation for #2342 .
I'm planning to parse JSON to use D3.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17442957
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala ---
@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
private[ui] class EnvironmentPage(parent: EnvironmentTab) extends WebUIPage("") {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+ val jvmInfoJson =
+ ("RUntime Informationf" -> listener.jvmInformation.foldLeft(JObject())(_ ~ _))
--- End diff --
There are a lot of typos in this line...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: (WIP) [SPARK-3454] Expose JSON representation ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-54952694
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20035/consoleFull) for PR 2333 at commit [`1cbf7d5`](https://github.com/apache/spark/commit/1cbf7d5e3583c41fc2bd20c77b73f204587b15f7).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443226
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -31,6 +36,35 @@ private[ui] class JobProgressPage(parent: JobProgressTab) extends WebUIPage("")
private val listener = parent.listener
private lazy val isFairScheduler = parent.isFairScheduler
+ override def renderJson(request: HttpServletRequest): JValue = {
+ listener.synchronized {
+
+ val activeStageList = listener.activeStages.values.map {
+ case info: StageInfo =>
+ JsonProtocol.stageInfoToJson(info)
+ }
--- End diff --
minor: just do
```
listener.activeStages.values.map { info => JsonProtocol.stageInfoToJson(info) }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55557935
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20326/consoleFull) for PR 2333 at commit [`d41b3ca`](https://github.com/apache/spark/commit/d41b3caf1adb0c807aa6ce9d011e5e2553408fe2).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class JavaSparkContext(val sc: SparkContext)`
* ` throw new IllegalStateException("The main method in the given main class must be static")`
* `class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception `
* ` class Dummy(object):`
* `class RatingDeserializer(FramedSerializer):`
* ` class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] `
* ` class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] `
* ` class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] `
* ` class Encoder extends compression.Encoder[IntegerType.type] `
* ` class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[IntegerType.type])`
* ` class Encoder extends compression.Encoder[LongType.type] `
* ` class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[LongType.type])`
* `class JavaStreamingContext(val ssc: StreamingContext) extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55974879
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20493/consoleFull) for PR 2333 at commit [`7b4d6eb`](https://github.com/apache/spark/commit/7b4d6eb92fbbf758854bf322608ed3ae2e68ebed).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443318
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -20,17 +20,76 @@ package org.apache.spark.ui.jobs
import java.util.Date
import javax.servlet.http.HttpServletRequest
+import org.json4s.JsonAST.JNothing
--- End diff --
minor, group this import with other `org.json4s` imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443156
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala ---
@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
private[ui] class EnvironmentPage(parent: EnvironmentTab) extends WebUIPage("") {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+ val jvmInfoJson =
+ ("RUntime Informationf" -> listener.jvmInformation.foldLeft(JObject())(_ ~ _))
+ val sparkPropertiesJson =
+ ("Spark Properties" -> listener.sparkProperties.foldLeft(JObject())(_ ~ _))
+ val systemPropertiesJson =
+ ("System Properties" -> listener.systemProperties.foldLeft(JObject())(_ ~ _))
+ val classPathEntriesJson =
+ ("Classpath Entries" -> listener.classpathEntries.foldLeft(JObject())(_ ~ _))
+
+ val environment = ("Environment" -> jvmInfoJson ~
+ sparkPropertiesJson ~
+ systemPropertiesJson ~
+ classPathEntriesJson)
--- End diff --
Should this just return the raw `*Json` things rather than keying it with `Environment`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-75205810
OK. I close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55678512
Can anyone review this if you have time?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55976178
@sarutak, I believe @JoshRosen is working on a more general framework for extracting the info displayed on the UI as JSON, so there's a chance that we won't go with the approach here. Just a heads up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55356808
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20200/consoleFull) for PR 2333 at commit [`72c0644`](https://github.com/apache/spark/commit/72c06445211579a207b36de2d4072974fe7325de).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443265
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -31,6 +36,35 @@ private[ui] class JobProgressPage(parent: JobProgressTab) extends WebUIPage("")
private val listener = parent.listener
private lazy val isFairScheduler = parent.isFairScheduler
+ override def renderJson(request: HttpServletRequest): JValue = {
+ listener.synchronized {
+
+ val activeStageList = listener.activeStages.values.map {
+ case info: StageInfo =>
+ JsonProtocol.stageInfoToJson(info)
+ }
+ val activeStageJson = ("Active Stages" -> activeStageList)
+
+ val completedStageList = listener.completedStages.reverse.map {
+ case info: StageInfo =>
+ JsonProtocol.stageInfoToJson(info)
+ }
+ val completedStageJson = ("Completed Stages" -> completedStageList)
+
+ val failedStageList = listener.failedStages.reverse.map {
+ case info: StageInfo =>
+ JsonProtocol.stageInfoToJson(info)
+ }
+ val failedStageJson = ("Failed Stages" -> failedStageList)
+
+ ("Stages Info" ->
+ ("Scheduling Mode" -> listener.schedulingMode.map(_.toString).getOrElse("Unknown")) ~
--- End diff --
Can you put this in a variable so this is more readable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: (WIP) [SPARK-3454] Expose JSON representation ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55237125
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20143/consoleFull) for PR 2333 at commit [`8850706`](https://github.com/apache/spark/commit/88507061c3c491af47831162a14763d685df3b83).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-75155273
@sarutak thanks a lot for working on this. There is another patch #4435 that provides a more elaborate implementation of the functionality here. Given that this PR has mostly gone stale at this point, would you mind closing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55967426
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20493/consoleFull) for PR 2333 at commit [`7b4d6eb`](https://github.com/apache/spark/commit/7b4d6eb92fbbf758854bf322608ed3ae2e68ebed).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: (WIP) [SPARK-3454] Expose JSON representation ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55243019
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20146/consoleFull) for PR 2333 at commit [`c108200`](https://github.com/apache/spark/commit/c1082008ee54aef85a023cb2551c8ca12e33feea).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55454772
@JoshRosen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55248376
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20146/consoleFull) for PR 2333 at commit [`c108200`](https://github.com/apache/spark/commit/c1082008ee54aef85a023cb2551c8ca12e33feea).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55514222
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20289/consoleFull) for PR 2333 at commit [`7f51a4f`](https://github.com/apache/spark/commit/7f51a4f1a979221ba8685662a66170c42f0447e9).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443332
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -20,17 +20,76 @@ package org.apache.spark.ui.jobs
import java.util.Date
import javax.servlet.http.HttpServletRequest
+import org.json4s.JsonAST.JNothing
+
import scala.xml.{Node, Unparsed}
+import org.json4s.{JObject, JValue}
+import org.json4s.JsonDSL._
+
import org.apache.spark.ui.{ToolTips, WebUIPage, UIUtils}
import org.apache.spark.ui.jobs.UIData._
-import org.apache.spark.util.{Utils, Distribution}
+import org.apache.spark.util.{JsonProtocol, Utils, Distribution}
import org.apache.spark.scheduler.AccumulableInfo
/** Page showing statistics and task list for a given stage */
private[ui] class StagePage(parent: JobProgressTab) extends WebUIPage("stage") {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+ val stageId = request.getParameter("id").toInt
+ val stageAttemptId = request.getParameter("attempt").toInt
+
+ var stageSummary = ("Stage ID" -> stageId) ~ ("Stage Attempt ID" -> stageAttemptId)
+
--- End diff --
extra new line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17461531
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala ---
@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
private[ui] class EnvironmentPage(parent: EnvironmentTab) extends WebUIPage("") {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+ val jvmInfoJson =
+ ("RUntime Informationf" -> listener.jvmInformation.foldLeft(JObject())(_ ~ _))
--- End diff --
This is the result of JSON for environment.
{
"Runtime Information" : {
"Java Home" : "/usr/java/jdk1.7.0_65/jre",
"Java Version" : "1.7.0_65 (Oracle Corporation)",
"Scala Version" : "version 2.10.4"
},
"Spark Properties" : {
"spark.serializer" : "org.apache.spark.serializer.KryoSerializer",
...
"spark.tachyonStore.folderName" : "spark-a8ac134b-187d-4d4b-b9ef-140e626ee96c"
},
"System Properties" : {
"java.io.tmpdir" : "/tmp",
...
"java.version" : "1.7.0_65",
"sun.io.unicode.encoding" : "UnicodeLittle"
},
"Classpath Entries" : {
"/home/sarutak/work/spark/sql/hive/target/scala-2.10/test-classes" : "System Classpath",
...
"/home/sarutak/work/spark/assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.2.0.jar" : "System Classpath"
}
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55253730
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20152/consoleFull) for PR 2333 at commit [`a349d0e`](https://github.com/apache/spark/commit/a349d0e26765174fd9147f6fced4693d5dc48c98).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55248298
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20152/consoleFull) for PR 2333 at commit [`a349d0e`](https://github.com/apache/spark/commit/a349d0e26765174fd9147f6fced4693d5dc48c98).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: (WIP) [SPARK-3454] Expose JSON representation ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55226767
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20141/consoleFull) for PR 2333 at commit [`f7958b0`](https://github.com/apache/spark/commit/f7958b04508e02d0f58895a3a8cc3e0b6fef33ab).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55977729
@sarutak In the long run, I'd be interested in re-writing the UI in terms of a richer REST API that exposes data as JSON, exactly for the visualization use-case that you mentioned. This change (Javascript-based rendering, etc) is blocked by a couple of other features right now. I'm swamped with a bunch of other work, but I'll try to get a public design draft posted in the next couple of days.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: (WIP) [SPARK-3454] Expose JSON representation ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2333#issuecomment-55226854
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20141/consoleFull) for PR 2333 at commit [`f7958b0`](https://github.com/apache/spark/commit/f7958b04508e02d0f58895a3a8cc3e0b6fef33ab).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org