You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by eatoncys <gi...@git.apache.org> on 2018/04/17 08:33:20 UTC
[GitHub] spark pull request #21084: [SPARK-23998][Core]It may be better to add @trans...
GitHub user eatoncys opened a pull request:
https://github.com/apache/spark/pull/21084
[SPARK-23998][Core]It may be better to add @transient to field 'taskMemoryManager' in class Task, for it is only be set and used in executor side
Add @transient to field 'taskMemoryManager' in class Task, for it is only be set and used in executor side and it will be set before used in class Executor like this:
task.setTaskMemoryManager(taskMemoryManager)
before task.run
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/eatoncys/spark transient
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21084.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21084
----
commit 283e1ac91996e375cb9d1775fef39ea29eb85325
Author: 10129659 <ch...@...>
Date: 2018-04-17T08:20:38Z
It may be better to add @transient to field 'taskMemoryManager' in class Task, for it is only be setted and used in executor
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21084
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21084
**[Test build #89439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89439/testReport)** for PR 21084 at commit [`283e1ac`](https://github.com/apache/spark/commit/283e1ac91996e375cb9d1775fef39ea29eb85325).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by eatoncys <gi...@git.apache.org>.
Github user eatoncys commented on the issue:
https://github.com/apache/spark/pull/21084
@jerryshao , There is not any issue without transient, but I think it is better to keep same to other fields, and make it clearly which fields do not need to be serialized.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by eatoncys <gi...@git.apache.org>.
Github user eatoncys commented on the issue:
https://github.com/apache/spark/pull/21084
@jerryshao @hvanhovell Ok, I will close it, thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21084
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21084
**[Test build #89439 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89439/testReport)** for PR 21084 at commit [`283e1ac`](https://github.com/apache/spark/commit/283e1ac91996e375cb9d1775fef39ea29eb85325).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21084
Seems OK to add `@transient`, but do you see any issue here without `transient`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21084
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2380/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by eatoncys <gi...@git.apache.org>.
Github user eatoncys commented on the issue:
https://github.com/apache/spark/pull/21084
@hvanhovell The field 'taskMemoryManager' is only used in executor side, so it is not needed to serialize it when sending the task from driver to executor
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/21084
@eatoncys what issue are you trying to solve?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21084
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89439/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/21084
Does it really takes a significant time to serialize the `taskMemoryManager` ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/21084
+0 as well. The downside to merging this is that it messes with the git blame.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by eatoncys <gi...@git.apache.org>.
Github user eatoncys commented on the issue:
https://github.com/apache/spark/pull/21084
@jiangxb1987 It does not take significant time to serialize the taskMemoryManager, because the value is null in driver side, but I think it is better to keep same to other fields in the Task class, which are only used in executor side, like '@volatile @transient private var _reasonIfKilled: String = null'.
In my test, the serialized size reduced from 8392 bytes to 8325 bytes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21084: [SPARK-23998][Core]It may be better to add @transient to...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21084
IIUC, null value will not be serialized (taskMemoryManager is only set in executor side), maybe Java will leave some footprints, but the overhead should be very small.
I'm +0 to fix, seems no harm to the codebase.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21084: [SPARK-23998][Core]It may be better to add @trans...
Posted by eatoncys <gi...@git.apache.org>.
Github user eatoncys closed the pull request at:
https://github.com/apache/spark/pull/21084
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org