You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/16 14:56:42 UTC
[GitHub] [spark] techaddict opened a new pull request, #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
techaddict opened a new pull request, #39093:
URL: https://github.com/apache/spark/pull/39093
**What changes were proposed in this pull request?**
Add Protobuf serializer for ApplicationInfoWrapper
**Why are the changes needed?**
Support fast and compact serialization/deserialization for ApplicationInfoWrapper over RocksDB.
**Does this PR introduce any user-facing change?**
No
**How was this patch tested?**
New UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] techaddict commented on pull request #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
Posted by GitBox <gi...@apache.org>.
techaddict commented on PR #39093:
URL: https://github.com/apache/spark/pull/39093#issuecomment-1356050497
@gengliangwang thanks for the review, updated the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a diff in pull request #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
Posted by GitBox <gi...@apache.org>.
mridulm commented on code in PR #39093:
URL: https://github.com/apache/spark/pull/39093#discussion_r1051302962
##########
core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto:
##########
@@ -106,3 +106,28 @@ message TaskDataWrapper {
int64 stage_id = 40;
int32 stage_attempt_id = 41;
}
+
+message ApplicationAttemptInfo {
+ optional string attempt_id = 1;
+ int64 start_time = 2;
+ int64 end_time = 3;
+ int64 last_updated = 4;
+ int64 duration = 5;
+ string spark_user = 6;
+ bool completed = 7;
+ string app_spark_version = 8;
+}
+
+message ApplicationInfo {
+ string id = 1;
+ string name = 2;
+ optional int32 cores_granted = 3;
+ optional int32 max_cores = 4;
+ optional int32 cores_per_executor = 5;
+ optional int32 memory_per_executor_mb = 6;
+ repeated ApplicationAttemptInfo attempts = 7;
+}
+
+message ApplicationInfoWrapper {
+ ApplicationInfo info = 1;
Review Comment:
My bad, ended up looking at the wrong file :) Thx for clarifying !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #39093:
URL: https://github.com/apache/spark/pull/39093#issuecomment-1358142702
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a diff in pull request #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
Posted by GitBox <gi...@apache.org>.
mridulm commented on code in PR #39093:
URL: https://github.com/apache/spark/pull/39093#discussion_r1051020503
##########
core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto:
##########
@@ -106,3 +106,28 @@ message TaskDataWrapper {
int64 stage_id = 40;
int32 stage_attempt_id = 41;
}
+
+message ApplicationAttemptInfo {
+ optional string attempt_id = 1;
+ int64 start_time = 2;
+ int64 end_time = 3;
+ int64 last_updated = 4;
+ int64 duration = 5;
+ string spark_user = 6;
+ bool completed = 7;
+ string app_spark_version = 8;
+}
+
+message ApplicationInfo {
+ string id = 1;
+ string name = 2;
+ optional int32 cores_granted = 3;
+ optional int32 max_cores = 4;
+ optional int32 cores_per_executor = 5;
+ optional int32 memory_per_executor_mb = 6;
+ repeated ApplicationAttemptInfo attempts = 7;
+}
+
+message ApplicationInfoWrapper {
+ ApplicationInfo info = 1;
Review Comment:
Missing `AttemptInfoWrapper` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] techaddict commented on a diff in pull request #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
Posted by GitBox <gi...@apache.org>.
techaddict commented on code in PR #39093:
URL: https://github.com/apache/spark/pull/39093#discussion_r1051244979
##########
core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto:
##########
@@ -106,3 +106,28 @@ message TaskDataWrapper {
int64 stage_id = 40;
int32 stage_attempt_id = 41;
}
+
+message ApplicationAttemptInfo {
+ optional string attempt_id = 1;
+ int64 start_time = 2;
+ int64 end_time = 3;
+ int64 last_updated = 4;
+ int64 duration = 5;
+ string spark_user = 6;
+ bool completed = 7;
+ string app_spark_version = 8;
+}
+
+message ApplicationInfo {
+ string id = 1;
+ string name = 2;
+ optional int32 cores_granted = 3;
+ optional int32 max_cores = 4;
+ optional int32 cores_per_executor = 5;
+ optional int32 memory_per_executor_mb = 6;
+ repeated ApplicationAttemptInfo attempts = 7;
+}
+
+message ApplicationInfoWrapper {
+ ApplicationInfo info = 1;
Review Comment:
@mridulm there are 2 `ApplicationInfoWrapper` one in `org.apache.spark.status`(this serializer is for that), and another one in `org.apache.spark.deploy.history`
both have different signatures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
URL: https://github.com/apache/spark/pull/39093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a diff in pull request #39093: [SPARK-41420][UI] Protobuf serializer for ApplicationInfoWrapper
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on code in PR #39093:
URL: https://github.com/apache/spark/pull/39093#discussion_r1051304099
##########
core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala:
##########
@@ -176,4 +176,59 @@ class KVStoreProtobufSerializerSuite extends SparkFunSuite {
assert(result.stageId == input.stageId)
assert(result.stageAttemptId == input.stageAttemptId)
}
+
+ test("Application Info") {
+ val attempts: Seq[ApplicationAttemptInfo] = Seq(
+ ApplicationAttemptInfo(
+ attemptId = Some("001"),
+ startTime = new Date(1),
+ endTime = new Date(10),
+ lastUpdated = new Date(10),
Review Comment:
Let's have different values in the test case to avoid mistakes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org