You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rkrzr <gi...@git.apache.org> on 2018/02/14 10:55:25 UTC
[GitHub] spark pull request #20607: Don't block on cleanup tasks by default
GitHub user rkrzr opened a pull request:
https://github.com/apache/spark/pull/20607
Don't block on cleanup tasks by default
This PR sets `"spark.cleaner.referenceTracking.blocking"` to `false` by
default. It had originally been set to `true` as a workaround for
SPARK-3015.
However, that issue has been resolved since 16/Aug/2014 already.
I would therefore think that it's safe to make this non-blocking by
default, which should help with cases where the cleanup thread can not
keep up any more.
If there are other reasons why this should stay blocking by default, I'd
be interested to learn about them. In that case the comment should
probably be updated.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/channable/spark dont_block_on_cleanup_tasks
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20607.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20607
----
commit f9e2f546b9b9beb136bb7110d0e4303365939927
Author: Robert Kreuzer <ro...@...>
Date: 2018-02-14T10:43:46Z
Don't block on cleanup tasks by default
This PR sets `"spark.cleaner.referenceTracking.blocking"` to false by
default. It had originally been set to true as a workaround for
SPARK-3015.
However, that issue has been resolved since 16/Aug/2014 already.
I would therefore think that it's safe to make this non-blocking by
default, which should help with cases where the cleanup thread can not
keep up any more.
If there are other reasons why this should stay blocking by default, I'd
be interested to learn about them. In that case the comment should
probably be updated.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20607: Don't block on cleanup tasks by default
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20607
Most probably nobody bothered to check.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20607: Don't block on cleanup tasks by default
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20607
@rkrzr please close the PR, unless you plan to actually test that this is not an issue anymore.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20607: Don't block on cleanup tasks by default
Posted by rkrzr <gi...@git.apache.org>.
Github user rkrzr commented on the issue:
https://github.com/apache/spark/pull/20607
I'll close this for now. (I am running into a problem related to this, but I'll better open a new issue about that when I know more)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20607: Don't block on cleanup tasks by default
Posted by rkrzr <gi...@git.apache.org>.
Github user rkrzr commented on the issue:
https://github.com/apache/spark/pull/20607
> SPARK-3015 was fixed by setting the config to true. So by setting it to false you'd be unfixing it, no?
Oh, my bad. You're right. I had only seen that it had been resolved, but not that this was the actual fix.
The [original issue](https://github.com/apache/spark/pull/1931/files) states that "This is intended to be only a temporary workaround for the issue". This was 3,5 years ago, so I wonder if the underlying issue has perhaps changed in the meantime?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20607: Don't block on cleanup tasks by default
Posted by rkrzr <gi...@git.apache.org>.
Github user rkrzr closed the pull request at:
https://github.com/apache/spark/pull/20607
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20607: Don't block on cleanup tasks by default
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20607
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20607: Don't block on cleanup tasks by default
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/20607
SPARK-3015 was fixed by setting the config to true. So by setting it to false you'd be unfixing it, no?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20607: Don't block on cleanup tasks by default
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20607
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org