You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jiangxb1987 <gi...@git.apache.org> on 2017/06/14 15:03:31 UTC
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/18303
[SPARK-19824][Core] Update JsonProtocol to keep consistent with the UI
## What changes were proposed in this pull request?
Fix any inconsistent part in JsonProtocol with the UI.
This PR also contains the modifications in #17181
## How was this patch tested?
Updated JsonProtocolSuite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark json-protocol
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18303.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18303
----
commit 244bbae71c2aa0b9f173ad7ac16ad0440eaab99c
Author: Xingbo Jiang <xi...@databricks.com>
Date: 2017-06-14T14:55:16Z
update JsonProtocol
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78227 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78227/testReport)** for PR 18303 at commit [`8dc5218`](https://github.com/apache/spark/commit/8dc52183f220da1bf2dcddcdae224943b62bc5eb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596647
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `user` name of the user who submitted the application
+ * `command` the command string that submitted the application
+ */
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
("name" -> obj.name) ~
- ("cores" -> obj.maxCores) ~
+ ("cores" -> obj.maxCores.getOrElse(0)) ~
("memoryperslave" -> obj.memoryPerExecutorMB) ~
("user" -> obj.user) ~
("command" -> obj.command.toString)
}
+ /**
+ * Export the [[ExecutorRunner]] to a Json object, an [[ExecutorRunner]] consists of the
+ * information of an executor.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a integer identifier of the executor
+ * `memory` memory in MB allocated to the executor
+ * `appid` a string identifier of the application that the executor is working for
--- End diff --
`working for` -> `working on`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18303
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78227 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78227/testReport)** for PR 18303 at commit [`8dc5218`](https://github.com/apache/spark/commit/8dc52183f220da1bf2dcddcdae224943b62bc5eb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596651
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `user` name of the user who submitted the application
+ * `command` the command string that submitted the application
+ */
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
("name" -> obj.name) ~
- ("cores" -> obj.maxCores) ~
+ ("cores" -> obj.maxCores.getOrElse(0)) ~
("memoryperslave" -> obj.memoryPerExecutorMB) ~
("user" -> obj.user) ~
("command" -> obj.command.toString)
}
+ /**
+ * Export the [[ExecutorRunner]] to a Json object, an [[ExecutorRunner]] consists of the
+ * information of an executor.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a integer identifier of the executor
+ * `memory` memory in MB allocated to the executor
+ * `appid` a string identifier of the application that the executor is working for
+ * `appdesc` a Json object of the [[ApplicationDescription]] of the application that the
+ * executor is working for
--- End diff --
`working for` -> `working on`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78154/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18303
Show the screenshot before and after this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596579
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
--- End diff --
-> the description of the application
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596663
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `user` name of the user who submitted the application
+ * `command` the command string that submitted the application
+ */
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
("name" -> obj.name) ~
- ("cores" -> obj.maxCores) ~
+ ("cores" -> obj.maxCores.getOrElse(0)) ~
("memoryperslave" -> obj.memoryPerExecutorMB) ~
("user" -> obj.user) ~
("command" -> obj.command.toString)
}
+ /**
+ * Export the [[ExecutorRunner]] to a Json object, an [[ExecutorRunner]] consists of the
+ * information of an executor.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a integer identifier of the executor
+ * `memory` memory in MB allocated to the executor
+ * `appid` a string identifier of the application that the executor is working for
+ * `appdesc` a Json object of the [[ApplicationDescription]] of the application that the
+ * executor is working for
+ */
def writeExecutorRunner(obj: ExecutorRunner): JObject = {
("id" -> obj.execId) ~
("memory" -> obj.memory) ~
("appid" -> obj.appId) ~
("appdesc" -> writeApplicationDescription(obj.appDesc))
}
+ /**
+ * Export the [[DriverInfo]] to a Json object, a [[DriverInfo]] consists of the information of a
--- End diff --
`, a` -> `. A`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78054/testReport)** for PR 18303 at commit [`244bbae`](https://github.com/apache/spark/commit/244bbae71c2aa0b9f173ad7ac16ad0440eaab99c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596343
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
--- End diff --
`, a` -> `. A`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18303
Thanks! Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122610140
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
--- End diff --
Good point, I'll remove this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18303
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596590
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
--- End diff --
`max cores can be ` -> `max cores that can be `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78054/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18303
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78227/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78235 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78235/testReport)** for PR 18303 at commit [`8c39912`](https://github.com/apache/spark/commit/8c399127741446b063c1b081593569bc76ad8fa8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596609
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `user` name of the user who submitted the application
+ * `command` the command string that submitted the application
+ */
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
("name" -> obj.name) ~
- ("cores" -> obj.maxCores) ~
+ ("cores" -> obj.maxCores.getOrElse(0)) ~
("memoryperslave" -> obj.memoryPerExecutorMB) ~
("user" -> obj.user) ~
("command" -> obj.command.toString)
}
+ /**
+ * Export the [[ExecutorRunner]] to a Json object, an [[ExecutorRunner]] consists of the
--- End diff --
`, an` -> `. An`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596599
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `user` name of the user who submitted the application
+ * `command` the command string that submitted the application
--- End diff --
`the command string that submitted` -> `the command submitted`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78054/testReport)** for PR 18303 at commit [`244bbae`](https://github.com/apache/spark/commit/244bbae71c2aa0b9f173ad7ac16ad0440eaab99c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596434
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
--- End diff --
If we already has host and port, why we still need the `address`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122384185
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -54,7 +55,7 @@ private[deploy] object JsonProtocol {
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
--- End diff --
Let me do this at the weekend in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18303
retest please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18303
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78163/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596562
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
--- End diff --
`, an` -> `. An`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18303
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78235 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78235/testReport)** for PR 18303 at commit [`8c39912`](https://github.com/apache/spark/commit/8c399127741446b063c1b081593569bc76ad8fa8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78163/testReport)** for PR 18303 at commit [`244bbae`](https://github.com/apache/spark/commit/244bbae71c2aa0b9f173ad7ac16ad0440eaab99c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596518
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
--- End diff --
Switch the order?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596533
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
--- End diff --
-> the description of the application
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18303
cc @yongtang @gatorsmile @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596741
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `user` name of the user who submitted the application
+ * `command` the command string that submitted the application
+ */
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
("name" -> obj.name) ~
- ("cores" -> obj.maxCores) ~
+ ("cores" -> obj.maxCores.getOrElse(0)) ~
("memoryperslave" -> obj.memoryPerExecutorMB) ~
("user" -> obj.user) ~
("command" -> obj.command.toString)
}
+ /**
+ * Export the [[ExecutorRunner]] to a Json object, an [[ExecutorRunner]] consists of the
+ * information of an executor.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a integer identifier of the executor
+ * `memory` memory in MB allocated to the executor
+ * `appid` a string identifier of the application that the executor is working for
+ * `appdesc` a Json object of the [[ApplicationDescription]] of the application that the
+ * executor is working for
+ */
def writeExecutorRunner(obj: ExecutorRunner): JObject = {
("id" -> obj.execId) ~
("memory" -> obj.memory) ~
("appid" -> obj.appId) ~
("appdesc" -> writeApplicationDescription(obj.appDesc))
}
+ /**
+ * Export the [[DriverInfo]] to a Json object, a [[DriverInfo]] consists of the information of a
+ * driver.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the driver
+ * `starttime` time in milliseconds that the driver starts
+ * `state` state of the driver, see [[DriverState]]
+ * `cores` cores allocated to the driver
+ * `memory` memory in MB allocated to the driver
+ * `submitdate` time in Date that the driver is created
+ * `worker` identifier of the worker that the driver is running on
+ * `mainclass` main class of the command string that started the driver
+ */
def writeDriverInfo(obj: DriverInfo): JObject = {
("id" -> obj.id) ~
("starttime" -> obj.startTime.toString) ~
("state" -> obj.state.toString) ~
("cores" -> obj.desc.cores) ~
- ("memory" -> obj.desc.mem)
+ ("memory" -> obj.desc.mem) ~
+ ("submitdate" -> obj.submitDate.toString) ~
+ ("worker" -> obj.worker.map(_.id).getOrElse("None")) ~
+ ("mainclass" -> obj.desc.command.arguments(2))
}
+ /**
+ * Export the [[MasterStateResponse]] to a Json object, a [[MasterStateResponse]] consists the
--- End diff --
`, a` -> `. A`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18303
LGTM except a few comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78235/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122377229
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -54,7 +55,7 @@ private[deploy] object JsonProtocol {
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
--- End diff --
IMO, we need to add the function descriptions for these public functions and provide a clear definition for each field. How about adding a few JIRAs and ask the open source community to do it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78154/testReport)** for PR 18303 at commit [`244bbae`](https://github.com/apache/spark/commit/244bbae71c2aa0b9f173ad7ac16ad0440eaab99c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18303
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596621
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `user` name of the user who submitted the application
+ * `command` the command string that submitted the application
+ */
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
("name" -> obj.name) ~
- ("cores" -> obj.maxCores) ~
+ ("cores" -> obj.maxCores.getOrElse(0)) ~
("memoryperslave" -> obj.memoryPerExecutorMB) ~
("user" -> obj.user) ~
("command" -> obj.command.toString)
}
+ /**
+ * Export the [[ExecutorRunner]] to a Json object, an [[ExecutorRunner]] consists of the
+ * information of an executor.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a integer identifier of the executor
--- End diff --
`a integer` -> `an integer`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18303
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78163/testReport)** for PR 18303 at commit [`244bbae`](https://github.com/apache/spark/commit/244bbae71c2aa0b9f173ad7ac16ad0440eaab99c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18303
**[Test build #78154 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78154/testReport)** for PR 18303 at commit [`244bbae`](https://github.com/apache/spark/commit/244bbae71c2aa0b9f173ad7ac16ad0440eaab99c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596570
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
--- End diff --
`, an` -> `. An`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18303: [SPARK-19824][Core] Update JsonProtocol to keep consiste...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18303
@gatorsmile Sure, let me do it later!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18303: [SPARK-19824][Core] Update JsonProtocol to keep c...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18303#discussion_r122596758
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -21,74 +21,192 @@ import org.json4s.JsonAST.JObject
import org.json4s.JsonDSL._
import org.apache.spark.deploy.DeployMessages.{MasterStateResponse, WorkerStateResponse}
-import org.apache.spark.deploy.master.{ApplicationInfo, DriverInfo, WorkerInfo}
+import org.apache.spark.deploy.master._
+import org.apache.spark.deploy.master.RecoveryState.MasterState
import org.apache.spark.deploy.worker.ExecutorRunner
private[deploy] object JsonProtocol {
- def writeWorkerInfo(obj: WorkerInfo): JObject = {
- ("id" -> obj.id) ~
- ("host" -> obj.host) ~
- ("port" -> obj.port) ~
- ("webuiaddress" -> obj.webUiAddress) ~
- ("cores" -> obj.cores) ~
- ("coresused" -> obj.coresUsed) ~
- ("coresfree" -> obj.coresFree) ~
- ("memory" -> obj.memory) ~
- ("memoryused" -> obj.memoryUsed) ~
- ("memoryfree" -> obj.memoryFree) ~
- ("state" -> obj.state.toString) ~
- ("lastheartbeat" -> obj.lastHeartbeat)
- }
+ /**
+ * Export the [[WorkerInfo]] to a Json object, a [[WorkerInfo]] consists of the information of a
+ * worker.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the worker
+ * `host` the host that the worker is running on
+ * `port` the port that the worker is bound to
+ * `address` ${host}:${port}
+ * `webuiaddress` the address used in web UI
+ * `cores` total cores of the worker
+ * `coresused` allocated cores of the worker
+ * `coresfree` free cores of the worker
+ * `memory` total memory of the worker
+ * `memoryused` allocated memory of the worker
+ * `memoryfree` free memory of the worker
+ * `state` state of the worker, see [[WorkerState]]
+ * `lastheartbeat` time in milliseconds that the latest heart beat message from the
+ * worker is received.
+ */
+ def writeWorkerInfo(obj: WorkerInfo): JObject = {
+ ("id" -> obj.id) ~
+ ("host" -> obj.host) ~
+ ("port" -> obj.port) ~
+ ("address" -> obj.hostPort) ~
+ ("webuiaddress" -> obj.webUiAddress) ~
+ ("cores" -> obj.cores) ~
+ ("coresused" -> obj.coresUsed) ~
+ ("coresfree" -> obj.coresFree) ~
+ ("memory" -> obj.memory) ~
+ ("memoryused" -> obj.memoryUsed) ~
+ ("memoryfree" -> obj.memoryFree) ~
+ ("state" -> obj.state.toString) ~
+ ("lastheartbeat" -> obj.lastHeartbeat)
+ }
+ /**
+ * Export the [[ApplicationInfo]] to a Json object, an [[ApplicationInfo]] consists of the
+ * information of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the application
+ * `starttime` time in milliseconds that the application starts
+ * `name` a name describes the application
+ * `cores` total cores granted to the application
+ * `user` name of the user who submitted the application
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `submitdate` time in Date that the application is submitted
+ * `state` state of the application, see [[ApplicationState]]
+ * `duration` time in milliseconds that the application has been running
+ */
def writeApplicationInfo(obj: ApplicationInfo): JObject = {
("starttime" -> obj.startTime) ~
("id" -> obj.id) ~
("name" -> obj.desc.name) ~
- ("cores" -> obj.desc.maxCores) ~
+ ("cores" -> obj.coresGranted) ~
("user" -> obj.desc.user) ~
("memoryperslave" -> obj.desc.memoryPerExecutorMB) ~
("submitdate" -> obj.submitDate.toString) ~
("state" -> obj.state.toString) ~
("duration" -> obj.duration)
}
+ /**
+ * Export the [[ApplicationDescription]] to a Json object, an [[ApplicationDescription]] consists
+ * of the description of an application.
+ *
+ * @return a Json object containing the following fields:
+ * `name` a name describes the application
+ * `cores` max cores can be allocated to the application, 0 means unlimited
+ * `memoryperslave` minimal memory in MB required to each executor
+ * `user` name of the user who submitted the application
+ * `command` the command string that submitted the application
+ */
def writeApplicationDescription(obj: ApplicationDescription): JObject = {
("name" -> obj.name) ~
- ("cores" -> obj.maxCores) ~
+ ("cores" -> obj.maxCores.getOrElse(0)) ~
("memoryperslave" -> obj.memoryPerExecutorMB) ~
("user" -> obj.user) ~
("command" -> obj.command.toString)
}
+ /**
+ * Export the [[ExecutorRunner]] to a Json object, an [[ExecutorRunner]] consists of the
+ * information of an executor.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a integer identifier of the executor
+ * `memory` memory in MB allocated to the executor
+ * `appid` a string identifier of the application that the executor is working for
+ * `appdesc` a Json object of the [[ApplicationDescription]] of the application that the
+ * executor is working for
+ */
def writeExecutorRunner(obj: ExecutorRunner): JObject = {
("id" -> obj.execId) ~
("memory" -> obj.memory) ~
("appid" -> obj.appId) ~
("appdesc" -> writeApplicationDescription(obj.appDesc))
}
+ /**
+ * Export the [[DriverInfo]] to a Json object, a [[DriverInfo]] consists of the information of a
+ * driver.
+ *
+ * @return a Json object containing the following fields:
+ * `id` a string identifier of the driver
+ * `starttime` time in milliseconds that the driver starts
+ * `state` state of the driver, see [[DriverState]]
+ * `cores` cores allocated to the driver
+ * `memory` memory in MB allocated to the driver
+ * `submitdate` time in Date that the driver is created
+ * `worker` identifier of the worker that the driver is running on
+ * `mainclass` main class of the command string that started the driver
+ */
def writeDriverInfo(obj: DriverInfo): JObject = {
("id" -> obj.id) ~
("starttime" -> obj.startTime.toString) ~
("state" -> obj.state.toString) ~
("cores" -> obj.desc.cores) ~
- ("memory" -> obj.desc.mem)
+ ("memory" -> obj.desc.mem) ~
+ ("submitdate" -> obj.submitDate.toString) ~
+ ("worker" -> obj.worker.map(_.id).getOrElse("None")) ~
+ ("mainclass" -> obj.desc.command.arguments(2))
}
+ /**
+ * Export the [[MasterStateResponse]] to a Json object, a [[MasterStateResponse]] consists the
+ * information of a master node.
+ *
+ * @return a Json object containing the following fields:
+ * `url` the url of the master node
+ * `workers` a list of Json objects of [[WorkerInfo]] of the workers allocated to the
+ * master
+ * `aliveworkers` size of alive workers allocated to the master
+ * `cores` total cores available of the master
+ * `coresused` cores used by the master
+ * `memory` total memory available of the master
+ * `memoryused` memory used by the master
+ * `activeapps` a list of Json objects of [[ApplicationInfo]] of the active applications
+ * running on the master
+ * `completedapps` a list of Json objects of [[ApplicationInfo]] of the completed
+ * applications from the master
--- End diff --
-> `the applications completed in the master`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org