You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ant.apache.org by "Ernest Pasour (JIRA)" <ji...@apache.org> on 2009/07/27 21:29:15 UTC
[jira] Created: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Excessive hits to missing URLs uses up all ports on Windows
-----------------------------------------------------------
Key: IVY-1105
URL: https://issues.apache.org/jira/browse/IVY-1105
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0.0-beta-2
Environment: windows xp sp2, linux fedora core 9, running Ivy repository through http for remote resolving and Hudson CI server (publishing to the repo).
Reporter: Ernest Pasour
Assignee: Maarten Coene
Fix For: 2.0-RC1
Ivy is extremely aggressive towards repositories . This can result in
resolving fails, even towards a healthy repository.
The symptom:
[ivy:resolve] 01-07-2008 13:16:24
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
[ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
processing request: Address already in use: connect.
In effect this happens when Ivy has performed a successfully DOS attack against the repository.
This is especially a problem when having large repositories (lost of revisions) and resolve against latest.status -> as this will fetch ivy.xml md5 and sha1 files for every revision.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ernest Pasour updated IVY-1105:
-------------------------------
Attachment: ApacheURLLister.java
hackaround that prevents the problem by capturing bad URLs and preventing them from being accessed again
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Attachments: ApacheURLLister.java
>
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Maarten Coene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maarten Coene resolved IVY-1105.
--------------------------------
Resolution: Fixed
Fix Version/s: trunk
ok, so I'll mark this issue as fixed.
Please reopen if you still have problems.
Thanks for the help in solving this issue!
Maarten
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Fix For: trunk
>
> Attachments: ApacheURLLister.java
>
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735731#action_12735731 ]
Ernest Pasour commented on IVY-1105:
------------------------------------
>From Nascif:
I believe we are seeing the same issue on Ivy 2.0 final, will post more details soon.
I have to agree that the current design is very much a DOS attack on the server. Our scenario is very much as described above as well, large number of modules, with large number of revisions, and resolving for latest. We can easily get to thousands of connections created and destroyed in a resolve, and we are bumping into the Windows limitations described in http://support.microsoft.com/default.aspx?scid=kb;EN-US;196271. For continuous integration builds, this is a major issue.
I wonder if a different approach could be taken for the url resolver. Instead of releasing the connection on every download() and upload(), why not take advantage of HTTP 1.1 ability to keep connections open? You would go from thousands of connections per resolve to 1 - and it would be constant, instead of on the order of modules x versions x resolvers as it is today.
Since there is already a concept of locking in Ivy, it means there is a concept of transaction borders. You could use that to implement to setup and closing of the connection.
http://java.sun.com/j2se/1.5.0/docs/guide/net/http-keepalive.html
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0.0-beta-2
> Environment: windows xp sp2, linux fedora core 9, running Ivy repository through http for remote resolving and Hudson CI server (publishing to the repo).
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Fix For: 2.0-RC1
>
>
> Ivy is extremely aggressive towards repositories . This can result in
> resolving fails, even towards a healthy repository.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> In effect this happens when Ivy has performed a successfully DOS attack against the repository.
> This is especially a problem when having large repositories (lost of revisions) and resolve against latest.status -> as this will fetch ivy.xml md5 and sha1 files for every revision.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735800#action_12735800 ]
Ernest Pasour commented on IVY-1105:
------------------------------------
It appears to be using BasicURLHandler. While I was debugging, I set breakpoints in HttpClientHandler and did not see them hit.
We have also reproduced this problem running from Ant (in the process of both doing builds), but it's a little more difficult to tell what's going on there.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735807#action_12735807 ]
Ernest Pasour commented on IVY-1105:
------------------------------------
Attached my hackaround. This suffers from general caching shortcomings, but has the side benefit of reducing unnecessary server connections.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Attachments: ApacheURLLister.java
>
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735732#action_12735732 ]
Ernest Pasour commented on IVY-1105:
------------------------------------
It appears that the problem we are experiencing right now is because of all the attempts to read non-existent directories using the ApacheURLLister class. When the connection attempts (and fails) to get the input stream, it is apparently enough to create a connection, but the connection is not cleaned up properly, even using the disconnect call. The code below reproduces the problem (on windows) if you put the proper data in the url definition. You will get the BindException after about 15 seconds. If you do a "netstat" from the command line, you will see lots of connections in the TIME_WAIT state.
I think Nascif's idea above makes sense to reduce the number of connections being made. I think it would also make sense to reduce the number of invalid URLs used. I hacked together some code to keep a hash of the shortest invalid urls and then not attempt to make connections when subsequent urls show up with an invalid prefix. This seems to work but I haven't really validated it yet. This idea could also be used in conjunction with Nascif's idea.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.BindException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
public class Test2 {
public static void main(String[] args) throws IOException {
for (int i=0;;i++)
{
URL url=new URL("http", "<your server>", 80, "<bad path on server>"); //shows problem
// URL url=new URL("http", "<your server>", 80, "<existing path on server>"); //works fine; does not leak connections
URLConnection conn=url.openConnection();
String htmlText = "";
try
{ BufferedReader r = new BufferedReader(new InputStreamReader(conn.getInputStream())); }
catch (BindException e)
{ e.printStackTrace(); }
catch (IOException e)
{}
finally
{
if (conn instanceof HttpURLConnection)
{ ((HttpURLConnection)conn).disconnect(); }
}
}
}
}
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0.0-beta-2
> Environment: windows xp sp2, linux fedora core 9, running Ivy repository through http for remote resolving and Hudson CI server (publishing to the repo).
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Fix For: 2.0-RC1
>
>
> Ivy is extremely aggressive towards repositories . This can result in
> resolving fails, even towards a healthy repository.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> In effect this happens when Ivy has performed a successfully DOS attack against the repository.
> This is especially a problem when having large repositories (lost of revisions) and resolve against latest.status -> as this will fetch ivy.xml md5 and sha1 files for every revision.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735788#action_12735788 ]
Ernest Pasour commented on IVY-1105:
------------------------------------
I already had httpclient 3.0 in my classpath. I updated to 3.1 and it didn't seem to make any difference. I think httpclient is on the classpath properly. I am running the Ivy source code (2.0) and using Ivy API methods directly if that makes a difference.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735801#action_12735801 ]
Ernest Pasour commented on IVY-1105:
------------------------------------
...but it does not appear that our Ant setup includes any version of http-client.jar. I'll have to check on that tomorrow.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Maarten Coene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735779#action_12735779 ]
Maarten Coene commented on IVY-1105:
------------------------------------
ok, I see the problem and I was able to fix the problem illustrated by your main-method, now I'll have to apply the same approach it to Ivy's codebase.
Can you confirm that this problem does *not* occur when you add commons-httpclient-3.x to Ivy's classpath?
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Maarten Coene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maarten Coene updated IVY-1105:
-------------------------------
Fix Version/s: 2.2.0-RC1
(was: trunk)
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Fix For: 2.2.0-RC1
>
> Attachments: ApacheURLLister.java
>
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735877#action_12735877 ]
Ernest Pasour commented on IVY-1105:
------------------------------------
That looks like it helps quite a bit! I now see perhaps one port being left around in TIME_WAIT state per resolve, which is much better than lots of ports being left around per resolve. I'll try to confirm tomorrow with a more taxing example.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Attachments: ApacheURLLister.java
>
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Maarten Coene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735795#action_12735795 ]
Maarten Coene commented on IVY-1105:
------------------------------------
Could you tell what URLHandler is used? For instance, what class does URLHandlerRegistry.getDefault() returns?
As far as I can tell, commons-httpclient will only be used if you use the Ant tasks or use the Ivy CLI.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Maarten Coene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maarten Coene updated IVY-1105:
-------------------------------
Affects Version/s: (was: 2.0)
2.1.0-RC1
Fix Version/s: (was: 2.0-RC1)
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Maarten Coene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735816#action_12735816 ]
Maarten Coene commented on IVY-1105:
------------------------------------
I've committed a potential fix in SVN trunk.
Could you please give it a try to see if it fixes the problem?
I couldn't test it because I'm on windows Vista right now which doens't seem to have the problem (butI could reproduce the problem earlier this day on a windows XP machine)
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Attachments: ApacheURLLister.java
>
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ernest Pasour updated IVY-1105:
-------------------------------
Environment: windows xp sp3, running Ivy repository through http for remote resolving (was: windows xp sp2, linux fedora core 9, running Ivy repository through http for remote resolving and Hudson CI server (publishing to the repo).)
Affects Version/s: (was: 2.0.0-beta-2)
2.0
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Fix For: 2.0-RC1
>
>
> Ivy is extremely aggressive towards repositories . This can result in
> resolving fails, even towards a healthy repository.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> In effect this happens when Ivy has performed a successfully DOS attack against the repository.
> This is especially a problem when having large repositories (lost of revisions) and resolve against latest.status -> as this will fetch ivy.xml md5 and sha1 files for every revision.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Jim Adams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735743#action_12735743 ]
Jim Adams commented on IVY-1105:
--------------------------------
I think some of the initial wording is a little odd. The idea that this is a DOS attack against the server is not right. In actuality it seems like a client performing a DOS attack on itself. In any case I think that portion of the wording is not right. The client is unusable until the socket times out.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Fix For: 2.0-RC1
>
>
> Ivy is extremely aggressive towards repositories . This can result in
> resolving fails, even towards a healthy repository.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> In effect this happens when Ivy has performed a successfully DOS attack against the repository.
> This is especially a problem when having large repositories (lost of revisions) and resolve against latest.status -> as this will fetch ivy.xml md5 and sha1 files for every revision.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ernest Pasour updated IVY-1105:
-------------------------------
Description:
Modified description to be clearer:
Ivy is extremely aggressive towards repositories . This can result in
failures to resolve, even against a healthy repository and with no network latency.
The symptom:
[ivy:resolve] 01-07-2008 13:16:24
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
[ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
processing request: Address already in use: connect.
This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
was:
Ivy is extremely aggressive towards repositories . This can result in
resolving fails, even towards a healthy repository.
The symptom:
[ivy:resolve] 01-07-2008 13:16:24
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
[ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
processing request: Address already in use: connect.
In effect this happens when Ivy has performed a successfully DOS attack against the repository.
This is especially a problem when having large repositories (lost of revisions) and resolve against latest.status -> as this will fetch ivy.xml md5 and sha1 files for every revision.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Fix For: 2.0-RC1
>
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (IVY-1105) Excessive hits to missing
URLs uses up all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735807#action_12735807 ]
Ernest Pasour edited comment on IVY-1105 at 7/27/09 3:12 PM:
-------------------------------------------------------------
Attached my hackaround. This suffers from general caching shortcomings, but has the side benefit of reducing unnecessary server connections. Minimally tested. I'm just including it here as a possibility if the underlying problem can't be fixed without a Java sdk fix.
was (Author: ernestp):
Attached my hackaround. This suffers from general caching shortcomings, but has the side benefit of reducing unnecessary server connections.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.1.0-RC1
> Environment: windows xp sp3, running Ivy repository through http for remote resolving
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Attachments: ApacheURLLister.java
>
>
> Modified description to be clearer:
> Ivy is extremely aggressive towards repositories . This can result in
> failures to resolve, even against a healthy repository and with no network latency.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.
> As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1105) Excessive hits to missing URLs uses up
all ports on Windows
Posted by "Ernest Pasour (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735730#action_12735730 ]
Ernest Pasour commented on IVY-1105:
------------------------------------
I've cloned https://issues.apache.org/jira/browse/IVY-854 because we appear to have the same issue, but I don't know how to change the status of it back to "Open". I'm copying the comments Nascif and I made on it into this comment for easier reference.
> Excessive hits to missing URLs uses up all ports on Windows
> -----------------------------------------------------------
>
> Key: IVY-1105
> URL: https://issues.apache.org/jira/browse/IVY-1105
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0.0-beta-2
> Environment: windows xp sp2, linux fedora core 9, running Ivy repository through http for remote resolving and Hudson CI server (publishing to the repo).
> Reporter: Ernest Pasour
> Assignee: Maarten Coene
> Fix For: 2.0-RC1
>
>
> Ivy is extremely aggressive towards repositories . This can result in
> resolving fails, even towards a healthy repository.
> The symptom:
> [ivy:resolve] 01-07-2008 13:16:24
> org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
> processing request: Address already in use: connect.
> In effect this happens when Ivy has performed a successfully DOS attack against the repository.
> This is especially a problem when having large repositories (lost of revisions) and resolve against latest.status -> as this will fetch ivy.xml md5 and sha1 files for every revision.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.