You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by d2r <gi...@git.apache.org> on 2017/03/31 22:00:31 UTC
[GitHub] spark pull request #17500: [SPARK-20181] [CORE] tries to bind the port to av...
GitHub user d2r opened a pull request:
https://github.com/apache/spark/pull/17500
[SPARK-20181] [CORE] tries to bind the port to avoid jetty noise
## What changes were proposed in this pull request?
Try to bind the desired port and release it before entering Jetty code, so there is less likelihood of hitting the BindException and thus seeing the warning log statement.
## How was this patch tested?
Manual Testing:
Before the change:
1. Launch spark-shell, configured to bind to port 4040 => observed successful bind to 4040
2. Launch spark-shell again, same config => WARN log & stack trace appear similar to Jira description, and port successfully binds to 4041.
After the change:
Repeat 1. and 2. above => No WARN log appears, and port successfully binds to 4041
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/d2r/spark spark-20181-supress-bindexception-stack
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17500.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17500
----
commit 180281fd80d9a581107d4a57535c26c3a01b2eca
Author: Derek Dagit <de...@yahoo-inc.com>
Date: 2017-03-31T21:11:16Z
tries to bind the port to avoid jetty noise
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17500
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17500: [SPARK-20181] [CORE] tries to bind the port to av...
Posted by d2r <gi...@git.apache.org>.
Github user d2r closed the pull request at:
https://github.com/apache/spark/pull/17500
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17500
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the issue:
https://github.com/apache/spark/pull/17500
> I get it, though I'm still not sure it's a great solution to artificially open and close the socket just for this. Is there no other way to probe whether a port is available or is that about the best one can do, to actually try to open it?
As far as I am aware, the best we can do is try to bind the port.
> log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
This works. There was an oversight on our part due to a merge error that we did not pick up this change when the jetty components were shaded. So I will close this pull request and close the jira as well.
@srowen Thanks for looking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17500
**[Test build #75576 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75576/testReport)** for PR 17500 at commit [`180281f`](https://github.com/apache/spark/commit/180281fd80d9a581107d4a57535c26c3a01b2eca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17500
Ping @d2r
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17500
**[Test build #75576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75576/testReport)** for PR 17500 at commit [`180281f`](https://github.com/apache/spark/commit/180281fd80d9a581107d4a57535c26c3a01b2eca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17500
Hm, why does this help? isn't it somehow trying to bind/close a port that's already in use?
What about just trying to reduce the log level or otherwise squelch the exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the issue:
https://github.com/apache/spark/pull/17500
@srowen
> Hm, why does this help? isn't it somehow trying to bind/close a port that's already in use?
If the port is in use, then it will throw a BindException here, which will be handled by logic in Spark code thereby avoiding logging a big stack trace in the Jetty code following it.
Without this change, if the port is in use, then Jetty code will log the exception with the full stack trace and then throw it.
In the case when the port is not used when entering this code, it is possible but less likely that we will see the stack trace. The scenario in which we would still see the stack trace is when the port is free before we enter the Jetty code, but then it becomes unavailable before Jetty tries to bind the port. It is a fairly tight race, but it could happen.
This change should hide the unhelpful stack trace a majority of the time.
> What about just trying to reduce the log level or otherwise squelch the exception?
We did look for a way to do this, but options are limited as the code is in Jetty.
Reducing the log level would be messy, and there are other log messages from Jetty at the WARN level that are very helpful. Unfortunately adjusting the log level dynamically here could hide useful information.
We actually do not want the exception squelched, we just want to avoid the log message.
Sorry for the late response. I need to fix my notification settings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17500
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75576/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17500
I get it, though I'm still not sure it's a great solution to artificially open and close the socket just for this. Is there no other way to probe whether a port is available or is that about the best one can do, to actually try to open it?
I see `log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle` in our log4j properties files, so clearly this is intended to be silenced by log config already. Is that not taking effect? if you haven't changed anything then clearly it isn't. I wonder if for some reason the unshaded package's logger needs to be configured, if so? `log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17500: [SPARK-20181] [CORE] tries to bind the port to avoid jet...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/17500
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org