You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mengxr <gi...@git.apache.org> on 2014/06/19 04:01:10 UTC
[GitHub] spark pull request: [SPARK-1112] use min akka frame size to decide...
GitHub user mengxr opened a pull request:
https://github.com/apache/spark/pull/1124
[SPARK-1112] use min akka frame size to decide how to send task results
Task results are sent either via akka directly or block manager indirectly, based on whether the size of the serialized task is smaller than spark.akka.frameSize. However, the result is actually sent back to the driver via the backend actor, which is initialized before receiving the SparkConf and hence it doesn't know the spark.akka.frameSize. That is the root cause of SPARK-1112.
A quick fix is using the min frame size to decide which route to go.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mengxr/spark akka-size
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1124.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1124
----
commit 5ca1edd0d01b66265d306e552c82b49d25537579
Author: Xiangrui Meng <me...@databricks.com>
Date: 2014-06-19T01:48:20Z
use min akka frame size to decide whether to send result via akka or block manager
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46525851
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46527422
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13955665
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -212,7 +208,12 @@ private[spark] class Executor(
val serializedDirectResult = ser.serialize(directResult)
logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)
val serializedResult = {
- if (serializedDirectResult.limit >= akkaFrameSize - 1024) {
+ // TODO: [SPARK-1112] We use the min frame size to determine whether to use Akka to send
+ // the task result or block manager. Since this is via the backend, whose actor system is
+ // initialized before receiving the Spark conf, and hence it does not know
+ // `spark.akka.frameSize`. A temporary solution is using the min frame size.
+ // [SPARK-2156] We subtract 200K to leave some space for other data in the Akka message.
+ if (serializedDirectResult.limit >= AkkaUtils.minFrameSizeBytes - 200 * 1024) {
--- End diff --
Re: your last question, I have no idea...you definitely know more about this than I do at this point. Using the higher, 200K value seems like a safe alternative to what's currently there. It would be great to add this constant in AkkaUtils so we don't need to manually track this down if it changes again in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46517699
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13955115
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -212,7 +208,12 @@ private[spark] class Executor(
val serializedDirectResult = ser.serialize(directResult)
logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)
val serializedResult = {
- if (serializedDirectResult.limit >= akkaFrameSize - 1024) {
+ // TODO: [SPARK-1112] We use the min frame size to determine whether to use Akka to send
+ // the task result or block manager. Since this is via the backend, whose actor system is
+ // initialized before receiving the Spark conf, and hence it does not know
+ // `spark.akka.frameSize`. A temporary solution is using the min frame size.
+ // [SPARK-2156] We subtract 200K to leave some space for other data in the Akka message.
+ if (serializedDirectResult.limit >= AkkaUtils.minFrameSizeBytes - 200 * 1024) {
--- End diff --
Why 200K and should we change the similar code that does this for sending task closures (which also subtracts 1024)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13951327
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -212,7 +208,14 @@ private[spark] class Executor(
val serializedDirectResult = ser.serialize(directResult)
logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)
val serializedResult = {
- if (serializedDirectResult.limit >= akkaFrameSize - 1024) {
+ // Akka's message frame size. If task result is bigger than this, we use the block manager
+ // to send the result back.
+ // TODO: [SPARK-1112] We use the min frame size to determine whether to use Akka to send
+ // TODO: the task result or block manager. Since this is via the backend, whose actor
+ // TODO: system is initialized before receiving the Spark conf, and hence it does not know
+ // TODO: `spark.akka.frameSize`. A temporary solution is using the min frame size.
+ // [SPARK-2156] We subtract 200K to leave some space for other data in the Akka message.
+ if (serializedDirectResult.limit >= AkkaUtils.minFrameSizeBytes - 200 * 1024) {
--- End diff --
`AkkaUtils.minFrameSizeBytes` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46525652
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112] use min akka frame size to decide...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46516262
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13952287
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -212,7 +208,14 @@ private[spark] class Executor(
val serializedDirectResult = ser.serialize(directResult)
logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)
val serializedResult = {
- if (serializedDirectResult.limit >= akkaFrameSize - 1024) {
+ // Akka's message frame size. If task result is bigger than this, we use the block manager
+ // to send the result back.
+ // TODO: [SPARK-1112] We use the min frame size to determine whether to use Akka to send
+ // TODO: the task result or block manager. Since this is via the backend, whose actor
--- End diff --
if you're here, would you mind changing this to just one "TODO:"... pls
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46530537
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112] use min akka frame size to decide...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46516744
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46517968
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46529444
So this can make performance worse for folks that set spark.akka.frameSize in the SparkConf that's on the executor to be higher, right, because this commit means we effectively ignore that parameter setting? Can we just figure out what the max frame size actually is for the backend actor system?
Also, this can lead to hung jobs if a user set the frame size to lower than the default of 10, right? @pwendell said you guys chose to ignore this case but it would be nice to add a comment explicitly stating this in the code in case it turns up as a bug later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13951554
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -212,7 +208,14 @@ private[spark] class Executor(
val serializedDirectResult = ser.serialize(directResult)
logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)
val serializedResult = {
- if (serializedDirectResult.limit >= akkaFrameSize - 1024) {
+ // Akka's message frame size. If task result is bigger than this, we use the block manager
+ // to send the result back.
+ // TODO: [SPARK-1112] We use the min frame size to determine whether to use Akka to send
+ // TODO: the task result or block manager. Since this is via the backend, whose actor
+ // TODO: system is initialized before receiving the Spark conf, and hence it does not know
+ // TODO: `spark.akka.frameSize`. A temporary solution is using the min frame size.
+ // [SPARK-2156] We subtract 200K to leave some space for other data in the Akka message.
+ if (serializedDirectResult.limit >= AkkaUtils.minFrameSizeBytes - 200 * 1024) {
--- End diff --
200 KB may not enough , Its value should increase as the serialized DirectResult becomes larger and larger.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112] use min akka frame size to decide...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46516257
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46519523
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15891/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46527961
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15899/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46530221
For the first scenario, it won't make the performance worse because the system doesn't really work now for serialized task result of size between 10M and spark.akka.frameSize. But yes, the ideal solution is to get the conf and set the right frame size. Maybe we can first request the conf from the driver, and then create a new ActorSystem on the backend with the correct frame size. This saves us from thinking about different deploy modes.
Any ActorSystem created using `AkkaUtils.createActorSystem` has at least 10M as the max frame size. It won't hang jobs.
Now I'm testing whether `- 1024` is risky in SchedulerBackend.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46518405
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46530539
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15896/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46517969
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15887/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46528519
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13952270
--- Diff: core/src/main/scala/org/apache/spark/util/AkkaUtils.scala ---
@@ -119,6 +119,9 @@ private[spark] object AkkaUtils extends Logging {
/** Returns the configured max frame size for Akka messages in bytes. */
def maxFrameSizeBytes(conf: SparkConf): Int = {
- conf.getInt("spark.akka.frameSize", 10) * 1024 * 1024
+ math.max(conf.getInt("spark.akka.frameSize", 0) * 1024 * 1024, minFrameSizeBytes)
}
+
+ /** The min frame size for Akka messages in bytes. */
--- End diff --
This is a slightly disingenuous name/comment, it's really the min maximum frame size.... Can keep the name, but maybe add some clarification in the comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46527960
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46531008
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46523174
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46531328
I don't get it. As long as the actor systems are created from AkkaUtils.createActorSystem, the minimum value of the max frame size is 10M. All unit tests passed on Jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46526053
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15895/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46525648
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13951639
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -212,7 +208,14 @@ private[spark] class Executor(
val serializedDirectResult = ser.serialize(directResult)
logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)
val serializedResult = {
- if (serializedDirectResult.limit >= akkaFrameSize - 1024) {
+ // Akka's message frame size. If task result is bigger than this, we use the block manager
+ // to send the result back.
+ // TODO: [SPARK-1112] We use the min frame size to determine whether to use Akka to send
+ // TODO: the task result or block manager. Since this is via the backend, whose actor
+ // TODO: system is initialized before receiving the Spark conf, and hence it does not know
+ // TODO: `spark.akka.frameSize`. A temporary solution is using the min frame size.
+ // [SPARK-2156] We subtract 200K to leave some space for other data in the Akka message.
+ if (serializedDirectResult.limit >= AkkaUtils.minFrameSizeBytes - 200 * 1024) {
--- End diff --
I tested with a double array of size very close to 10 * 1024 * 1024. The akka message overhead is about 30-60K. This PR doesn't fix the issues with receiving new tasks from the driver that are bigger than 10MB. @pwendell is working on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46525502
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15894/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46531010
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15902/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46525856
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46531776
Just tested `- 1024` in `SchedulerBackend`. The system did hang up when the task size is close to 10M - 1024 ...
~~~
scala> val random = new java.util.Random(0); val a = new Array[Byte](10 * 1024 * 1024 - 110000); random.nextBytes(a); sc.parallelize(0 until 1, 1).map(i => a).count()
14/06/19 00:33:59 INFO SparkContext: Starting job: count at <console>:14
14/06/19 00:33:59 INFO DAGScheduler: Got job 5 (count at <console>:14) with 1 output partitions (allowLocal=false)
14/06/19 00:33:59 INFO DAGScheduler: Final stage: Stage 5(count at <console>:14)
14/06/19 00:33:59 INFO DAGScheduler: Parents of final stage: List()
14/06/19 00:33:59 INFO DAGScheduler: Missing parents: List()
14/06/19 00:33:59 INFO DAGScheduler: Submitting Stage 5 (MappedRDD[11] at map at <console>:14), which has no missing parents
14/06/19 00:34:00 INFO DAGScheduler: Submitting 1 missing tasks from Stage 5 (MappedRDD[11] at map at <console>:14)
14/06/19 00:34:00 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
14/06/19 00:34:00 INFO TaskSetManager: Starting task 5.0:0 as TID 5 on executor 0: xm.att.net (PROCESS_LOCAL)
14/06/19 00:34:00 INFO TaskSetManager: Serialized task 5.0:0 as 10431605 bytes in 214 ms
task size: 10482813
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46528231
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13955317
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -212,7 +208,12 @@ private[spark] class Executor(
val serializedDirectResult = ser.serialize(directResult)
logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)
val serializedResult = {
- if (serializedDirectResult.limit >= akkaFrameSize - 1024) {
+ // TODO: [SPARK-1112] We use the min frame size to determine whether to use Akka to send
+ // the task result or block manager. Since this is via the backend, whose actor system is
+ // initialized before receiving the Spark conf, and hence it does not know
+ // `spark.akka.frameSize`. A temporary solution is using the min frame size.
+ // [SPARK-2156] We subtract 200K to leave some space for other data in the Akka message.
+ if (serializedDirectResult.limit >= AkkaUtils.minFrameSizeBytes - 200 * 1024) {
--- End diff --
@kayousterhout I saw the line in `CoarseGrainedSchedulerBackend`. Should the overhead be bounded by a fixed size or proportional to the message size?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46531448
I see -- you're right about the min frame size. @pwendell was talking about the first case, where I think the config parameter is currently being correctly honored
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46528511
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46769996
Closing this in favor of #1132.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46530520
@mengxr this actually hung up while running tests:
```
[info] MapOutputTrackerSuite:
[info] - compressSize
[info] - decompressSize
[info] - master start and stop
[info] - master register shuffle and fetch
[info] - master register and unregister shuffle
[info] - master register shuffle and unregister map output and fetch
[info] - remote fetch
[info] - remote fetch below akka frame size
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46527308
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46530379
@mengxr I think the scenario is where people have set SPARK_JAVA_OPTS in `spark-env` and copied this to all of the nodes. In that case, the configuration will be correctly observed by the actor system in `CoarseGrainedExectorBackend`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46518406
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15888/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46517694
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112] use min akka frame size to decide...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46516396
Is there a use case for making the min frame size configurable with a parameter?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46519521
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1124#discussion_r13955220
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -212,7 +208,12 @@ private[spark] class Executor(
val serializedDirectResult = ser.serialize(directResult)
logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)
val serializedResult = {
- if (serializedDirectResult.limit >= akkaFrameSize - 1024) {
+ // TODO: [SPARK-1112] We use the min frame size to determine whether to use Akka to send
+ // the task result or block manager. Since this is via the backend, whose actor system is
+ // initialized before receiving the Spark conf, and hence it does not know
+ // `spark.akka.frameSize`. A temporary solution is using the min frame size.
+ // [SPARK-2156] We subtract 200K to leave some space for other data in the Akka message.
+ if (serializedDirectResult.limit >= AkkaUtils.minFrameSizeBytes - 200 * 1024) {
--- End diff --
We computed the difference between the size of the akka message and the size of serialized task result (~10M). The difference is smaller than 60K. I set 200K to be safe. Could you point me to the places where we use 1024?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr closed the pull request at:
https://github.com/apache/spark/pull/1124
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112] use min akka frame size to decide...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46516754
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46517676
@ash211 If there is a way to make the configuration delivered consistently to backend, we can use `spark.akka.frameSize` consistently. It is then not necessary to set the min frame size.
This is a quick fix for `0.9.2` and `1.0.1`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46525500
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46518488
I'm not sure the implication of it, but it sounds like it's probably ok to
hard code this value for the quick fix. Thanks!
On Wed, Jun 18, 2014 at 7:34 PM, Xiangrui Meng <no...@github.com>
wrote:
> @ash211 <https://github.com/ash211> If there is a way to make the
> configuration delivered consistently to backend, we can use
> spark.akka.frameSize consistently. It is then not necessary to set the
> min frame size.
>
> This is a quick fix for 0.9.2 and 1.0.1.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/1124#issuecomment-46517676>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46526052
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46523179
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46520703
LGTM. Note that this fix is "safe" in that in the most conservative case, we'll just return the result through the block manager rather than Akka. This may be a little slower, but should be guaranteed to work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46541699
@pwendell @kayousterhout I put an alternative solution in #1132 . Please let me know which do you prefer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1112, 2156] use min akka frame size to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1124#issuecomment-46527415
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---