You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Krzysztof Zimnicki <kr...@go2.pl> on 2007/12/04 22:58:20 UTC
Threads problem - too many errors.
Hello.
My application using threads, do 1000-10000 conections. I create 20-30
threads and at begining everything is fine, but 2-3h later i have only
50% works threads. I don't know what is wrong, and have more errors than
at begining. My code:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
import org.apache.commons.httpclient.URI;
import org.apache.commons.httpclient.methods.GetMethod;
public class SingleCheck extends Thread {
private static BufferedReader fileUrl = null;
private HttpClient client = null;
private GetMethod get = null;
public SingleCheck(String id)
throws FileNotFoundException {
super(id);
System.out.println(id);
client = new HttpClient(new MultiThreadedHttpConnectionManager());
client.getHttpConnectionManager().getParams().setConnectionTimeout(
10000);
get = new GetMethod();
if (fileUrl == null) {
fileUrl = new BufferedReader(new FileReader(new File(
"URLS.txt")));
}
}
synchronized String pobierzURL() throws IOException {
return fileUrl.readLine();
}
boolean check(String url) throws HttpException,
IOException {
try {
URI uri = new URI(url, true);
get.setURI(uri);
get.addRequestHeader("Connection", "close");
client.executeMethod(get);
Header[] headers = get.getResponseHeaders();
for (int i = 0; i < headers.length; i++) {
if (headers[i].getName().contains(("Location"))) {
return true;
}
}
return false;
} finally {
get.releaseConnection();
}
}
public void run() {
String url = "";
while (url != null) {
try {
url = this.pobierzURL();
System.out.println("Watek nr: " + getName() + " " + url);
if (this.check( url)) {
//add to file if url is good
ZarzadzanieWynikami.addToFile( url,"goodUrls.txt");
}
} catch (Exception e) {
e.printStackTrace();
System.out.println("Exception catch");
}
}
}
}
and main file:
public class MainLoginCheck {
/**
* @param args
*/
public static void main(String[] args) {
try {
System.out.println(args[1]);
int iloscWatkow = Integer.parseInt(args[0]);
String haslo = args[1];
for (int i = 0; i < iloscWatkow; i++) {
new SingleCheck( "" + i).start();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Many of errors:
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at SingleCheck.login(SingleCheck.java:61)
at SingleCheck.run(SingleCheck.java:90)
Any idea?
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Threads problem - too many errors.
Posted by Krzysztof Zimnicki <kr...@go2.pl>.
>
> The connection manager will automatically allocate new connections as
> necessary, but you need to make sure to restart the worker threads that
> terminated due to the SocketException.
>
> Oleg
>
>
Thanks, i try do this.
Krzysztof Zimnicki
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Threads problem - too many errors.
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2007-12-06 at 17:56 +0100, Krzysztof Zimnicki wrote:
> Oleg Kalnichevski pisze:
> > On Tue, 2007-12-04 at 22:58 +0100, Krzysztof Zimnicki wrote:
> >
> >> Hello.
> >>
> >> My application using threads, do 1000-10000 conections. I create 20-30
> >> threads and at begining everything is fine, but 2-3h later i have only
> >> 50% works threads. I don't know what is wrong, and have more errors than
> >> at begining. My code:
> >>
> >>
> >
> >
> >> Many of errors:
> >>
> >> java.net.SocketException: Connection reset
> >> at java.net.SocketInputStream.read(Unknown Source)
> >>
> >
> > ...
> >
> >
> >> Any idea?
> >>
> >>
> >
> > Krzysztof,
> >
> > 'SocketException: Connection reset' can happen. Your application should
> > compensate for failed connections by opening new ones.
> >
> > Oleg
> >
> >
> >
> How can i compesate for failed connections ? releaseConnection(); don't
> do that ?
> Why? Example please :)
>
The connection manager will automatically allocate new connections as
necessary, but you need to make sure to restart the worker threads that
terminated due to the SocketException.
Oleg
> Krzysztof Zimnicki
>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >>
> >>
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Threads problem - too many errors.
Posted by Krzysztof Zimnicki <kr...@go2.pl>.
Oleg Kalnichevski pisze:
> On Tue, 2007-12-04 at 22:58 +0100, Krzysztof Zimnicki wrote:
>
>> Hello.
>>
>> My application using threads, do 1000-10000 conections. I create 20-30
>> threads and at begining everything is fine, but 2-3h later i have only
>> 50% works threads. I don't know what is wrong, and have more errors than
>> at begining. My code:
>>
>>
>
>
>> Many of errors:
>>
>> java.net.SocketException: Connection reset
>> at java.net.SocketInputStream.read(Unknown Source)
>>
>
> ...
>
>
>> Any idea?
>>
>>
>
> Krzysztof,
>
> 'SocketException: Connection reset' can happen. Your application should
> compensate for failed connections by opening new ones.
>
> Oleg
>
>
>
How can i compesate for failed connections ? releaseConnection(); don't
do that ?
Why? Example please :)
Krzysztof Zimnicki
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Threads problem - too many errors.
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2007-12-04 at 22:58 +0100, Krzysztof Zimnicki wrote:
> Hello.
>
> My application using threads, do 1000-10000 conections. I create 20-30
> threads and at begining everything is fine, but 2-3h later i have only
> 50% works threads. I don't know what is wrong, and have more errors than
> at begining. My code:
>
> Many of errors:
>
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(Unknown Source)
...
>
> Any idea?
>
Krzysztof,
'SocketException: Connection reset' can happen. Your application should
compensate for failed connections by opening new ones.
Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org