You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2012/07/31 23:37:14 UTC
NullPointerException in BestMatchSpec.formatCookies
Hi,
I'm getting the exception below when I'm trying to load
http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
... It's not critical since I'm catching everything, but I'm wondering
if this is normal or if this is something which need to be fixed...
java.lang.NullPointerException
at org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
--
JM
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: NullPointerException in BestMatchSpec.formatCookies
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
So it's can be that neither... Thanks for the links.
I have restarted the application. I have added the debug level for
RequestAddCookies and asked the application to stop as soon as there
is an exception. I just need to let it run for some time. Last time I
saw the exception it was running for few days. So it might take time
before it happens again.
2012/8/2, sebb <se...@gmail.com>:
> On 2 August 2012 19:36, Jean-Marc Spaggiari <je...@spaggiari.org>
> wrote:
>> I'm using 1.7.0_05 from Sun.
>>
>> I have enabled the RequestAddCookies debug logs and will see when I
>> will get the next NPE.
>>
>> One the line 193 (I don't have the source) is version an int? Or an
>> Integer?
>
> int, see:
>
> http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/impl/cookie/BestMatchSpec.html#193
> and
> http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/cookie/Cookie.html#126
>
>> I agree with you, cookie can't be null, else we will have got an NPE
>> at "if (!matchedCookies.isEmpty())"
>>
>> I will restart my tool with the debug mode enabled and keep you posted.
>>
>> 2012/8/2, sebb <se...@gmail.com>:
>>> On 2 August 2012 19:03, Jean-Marc Spaggiari <je...@spaggiari.org>
>>> wrote:
>>>> I'm using version 4.2.1 binary distribution.
>>>
>>> In which case the NPE is very odd.
>>>
>>> The stack dump shows:
>>>
>>> java.lang.NullPointerException
>>> at
>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>> at
>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>>
>>> Now the code in question is:
>>>
>>> BestMatchSpec.java:193: if (cookie.getVersion() < version) {
>>>
>>> AFAICT, this can only cause NPE if "cookie" is null.
>>>
>>> The cookie variable is derived from the list created in
>>> RequestAddCookies.java:
>>>
>>> List<Cookie> cookies = new
>>> ArrayList<Cookie>(cookieStore.getCookies());
>>> // Find cookies matching the given origin
>>> List<Cookie> matchedCookies = new ArrayList<Cookie>();
>>> Date now = new Date();
>>> for (Cookie cookie : cookies) {
>>> if (!cookie.isExpired(now)) { // <== cookie cannot be null
>>> here
>>> if (cookieSpec.match(cookie, cookieOrigin)) {
>>> if (this.log.isDebugEnabled()) {
>>> this.log.debug("Cookie " + cookie + " match "
>>> + cookieOrigin);
>>> }
>>> matchedCookies.add(cookie); // <== so cannot be null
>>> here
>>> }
>>> } else {
>>> if (this.log.isDebugEnabled()) {
>>> this.log.debug("Cookie " + cookie + " expired");
>>> }
>>> }
>>> }
>>> // Generate Cookie request headers
>>> if (!matchedCookies.isEmpty()) {
>>> List<Header> headers =
>>> cookieSpec.formatCookies(matchedCookies); // <== 196
>>> for (Header header : headers) {
>>> request.addHeader(header);
>>> }
>>> }
>>>
>>> I cannot see how a null cookie can be added to the list, nor how a
>>> list entry could be set to null later.
>>>
>>> It would be worth enabling Debug logging for the RequestAddCookies
>>> class.
>>>
>>> What JVM are you using?
>>>
>>>> 2012/8/2, sebb <se...@gmail.com>:
>>>>> On 1 August 2012 12:08, Jean-Marc Spaggiari <je...@spaggiari.org>
>>>>> wrote:
>>>>>> Hi Oleg,
>>>>>>
>>>>>> My application is reading many other URLs before this one. So maybe
>>>>>> it's a combination of multiple loads.
>>>>>>
>>>>>> I will try to dump more when I will get the issue again.
>>>>>>
>>>>>> In the meantime, here is how I'm initializing the HttpClient:
>>>>>>
>>>>>> // Creation of the HTTP Client
>>>>>> SchemeRegistry schemeRegistry = new SchemeRegistry();
>>>>>> schemeRegistry.register(new Scheme("http", 80,
>>>>>> PlainSocketFactory.getSocketFactory()));
>>>>>> schemeRegistry.register(new Scheme("https", 443,
>>>>>> SSLSocketFactory.getSocketFactory()));
>>>>>> PoolingClientConnectionManager cm = new
>>>>>> PoolingClientConnectionManager(schemeRegistry);
>>>>>> cm.setMaxTotal(200);
>>>>>> cm.setDefaultMaxPerRoute(20);
>>>>>>
>>>>>> HttpParams params = new BasicHttpParams();
>>>>>> if (useProxy)
>>>>>> params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new
>>>>>> HttpHost("proxy", 80));
>>>>>> client = new DefaultHttpClient(cm, params);
>>>>>> client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, new
>>>>>> Integer(30000));
>>>>>> client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
>>>>>> new Integer(30000));
>>>>>> client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY,
>>>>>> false);
>>>>>>
>>>>>> It's not the first time I'm getting this exception, so I might be
>>>>>> able
>>>>>> to reproduce that in a near futur.
>>>>>
>>>>> What version of HttpClient are you using?
>>>>>
>>>>>> JM
>>>>>>
>>>>>> 2012/8/1, Oleg Kalnichevski <ol...@apache.org>:
>>>>>>> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm getting the exception below when I'm trying to load
>>>>>>>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
>>>>>>>> ... It's not critical since I'm catching everything, but I'm
>>>>>>>> wondering
>>>>>>>> if this is normal or if this is something which need to be fixed...
>>>>>>>>
>>>>>>>> java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>>>>>>> at
>>>>>>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>>>>>>> at
>>>>>>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
>>>>>>>> at
>>>>>>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>>>>>>>> at
>>>>>>>> org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> JM
>>>>>>>>
>>>>>>>
>>>>>>> Jean-Marc
>>>>>>>
>>>>>>> I am not able to reproduce the problem just by executing HTTP GET to
>>>>>>> the
>>>>>>> URL in question. All looks perfectly normal.
>>>>>>>
>>>>>>> [DEBUG] BasicClientConnectionManager - Get connection for route
>>>>>>> {}->http://www.expedia.ca
>>>>>>> [DEBUG] DefaultClientConnectionOperator - Connecting to
>>>>>>> www.expedia.ca:80
>>>>>>> [DEBUG] RequestAddCookies - CookieSpec selected: best-match
>>>>>>> [DEBUG] RequestAuthCache - Auth cache not set in the context
>>>>>>> [DEBUG] RequestTargetAuthentication - Target auth state:
>>>>>>> UNCHALLENGED
>>>>>>> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
>>>>>>> [DEBUG] DefaultHttpClient - Attempt 1 to execute request
>>>>>>> [DEBUG] DefaultClientConnection - Sending request:
>>>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>>>>> [DEBUG] headers - >>
>>>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>>>>> [DEBUG] headers - >> Host: www.expedia.ca
>>>>>>> [DEBUG] headers - >> Connection: Keep-Alive
>>>>>>> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
>>>>>>> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200
>>>>>>> OK
>>>>>>> [DEBUG] headers - << HTTP/1.1 200 OK
>>>>>>> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8
>>>>>>> [DEBUG] headers - << Content-Language: en-CA
>>>>>>> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND
>>>>>>> COR
>>>>>>> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
>>>>>>> [DEBUG] headers - << RTSS: 1
>>>>>>> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
>>>>>>> [DEBUG] headers - << Transfer-Encoding: chunked
>>>>>>> [DEBUG] headers - << Connection: keep-alive
>>>>>>> [DEBUG] headers - << Connection: Transfer-Encoding
>>>>>>> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
>>>>>>> [DEBUG] headers - << Set-Cookie:
>>>>>>> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
>>>>>>> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
>>>>>>> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
>>>>>>> Path=/
>>>>>>> [DEBUG] headers - << Set-Cookie:
>>>>>>> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
>>>>>>> Path=/
>>>>>>> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/
>>>>>>> [DEBUG] headers - << Set-Cookie:
>>>>>>> p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
>>>>>>> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
>>>>>>> 01-Aug-2017 15:54:40 GMT; Path=/
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> MC1][value:
>>>>>>> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path:
>>>>>>> /][expiry:
>>>>>>> Fri Aug 04 04:44:09 GMT+01:00 2017]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry:
>>>>>>> null]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> JSESSION][value:
>>>>>>> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path:
>>>>>>> /][expiry:
>>>>>>> null]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
>>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version:
>>>>>>> 0][name:
>>>>>>> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
>>>>>>> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01
>>>>>>> 16:54:40
>>>>>>> GMT+01:00 2017]".
>>>>>>> [DEBUG] DefaultHttpClient - Connection can be kept alive
>>>>>>> indefinitely
>>>>>>>
>>>>>>> Can you put together a small test app that could be used to
>>>>>>> reproduce
>>>>>>> the NPE?
>>>>>>>
>>>>>>> Oleg
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: NullPointerException in BestMatchSpec.formatCookies
Posted by sebb <se...@gmail.com>.
On 2 August 2012 19:36, Jean-Marc Spaggiari <je...@spaggiari.org> wrote:
> I'm using 1.7.0_05 from Sun.
>
> I have enabled the RequestAddCookies debug logs and will see when I
> will get the next NPE.
>
> One the line 193 (I don't have the source) is version an int? Or an Integer?
int, see:
http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/impl/cookie/BestMatchSpec.html#193
and
http://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/apache/http/cookie/Cookie.html#126
> I agree with you, cookie can't be null, else we will have got an NPE
> at "if (!matchedCookies.isEmpty())"
>
> I will restart my tool with the debug mode enabled and keep you posted.
>
> 2012/8/2, sebb <se...@gmail.com>:
>> On 2 August 2012 19:03, Jean-Marc Spaggiari <je...@spaggiari.org>
>> wrote:
>>> I'm using version 4.2.1 binary distribution.
>>
>> In which case the NPE is very odd.
>>
>> The stack dump shows:
>>
>> java.lang.NullPointerException
>> at
>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>> at
>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>
>> Now the code in question is:
>>
>> BestMatchSpec.java:193: if (cookie.getVersion() < version) {
>>
>> AFAICT, this can only cause NPE if "cookie" is null.
>>
>> The cookie variable is derived from the list created in
>> RequestAddCookies.java:
>>
>> List<Cookie> cookies = new
>> ArrayList<Cookie>(cookieStore.getCookies());
>> // Find cookies matching the given origin
>> List<Cookie> matchedCookies = new ArrayList<Cookie>();
>> Date now = new Date();
>> for (Cookie cookie : cookies) {
>> if (!cookie.isExpired(now)) { // <== cookie cannot be null here
>> if (cookieSpec.match(cookie, cookieOrigin)) {
>> if (this.log.isDebugEnabled()) {
>> this.log.debug("Cookie " + cookie + " match "
>> + cookieOrigin);
>> }
>> matchedCookies.add(cookie); // <== so cannot be null
>> here
>> }
>> } else {
>> if (this.log.isDebugEnabled()) {
>> this.log.debug("Cookie " + cookie + " expired");
>> }
>> }
>> }
>> // Generate Cookie request headers
>> if (!matchedCookies.isEmpty()) {
>> List<Header> headers =
>> cookieSpec.formatCookies(matchedCookies); // <== 196
>> for (Header header : headers) {
>> request.addHeader(header);
>> }
>> }
>>
>> I cannot see how a null cookie can be added to the list, nor how a
>> list entry could be set to null later.
>>
>> It would be worth enabling Debug logging for the RequestAddCookies class.
>>
>> What JVM are you using?
>>
>>> 2012/8/2, sebb <se...@gmail.com>:
>>>> On 1 August 2012 12:08, Jean-Marc Spaggiari <je...@spaggiari.org>
>>>> wrote:
>>>>> Hi Oleg,
>>>>>
>>>>> My application is reading many other URLs before this one. So maybe
>>>>> it's a combination of multiple loads.
>>>>>
>>>>> I will try to dump more when I will get the issue again.
>>>>>
>>>>> In the meantime, here is how I'm initializing the HttpClient:
>>>>>
>>>>> // Creation of the HTTP Client
>>>>> SchemeRegistry schemeRegistry = new SchemeRegistry();
>>>>> schemeRegistry.register(new Scheme("http", 80,
>>>>> PlainSocketFactory.getSocketFactory()));
>>>>> schemeRegistry.register(new Scheme("https", 443,
>>>>> SSLSocketFactory.getSocketFactory()));
>>>>> PoolingClientConnectionManager cm = new
>>>>> PoolingClientConnectionManager(schemeRegistry);
>>>>> cm.setMaxTotal(200);
>>>>> cm.setDefaultMaxPerRoute(20);
>>>>>
>>>>> HttpParams params = new BasicHttpParams();
>>>>> if (useProxy)
>>>>> params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new
>>>>> HttpHost("proxy", 80));
>>>>> client = new DefaultHttpClient(cm, params);
>>>>> client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, new
>>>>> Integer(30000));
>>>>> client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
>>>>> new Integer(30000));
>>>>> client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY,
>>>>> false);
>>>>>
>>>>> It's not the first time I'm getting this exception, so I might be able
>>>>> to reproduce that in a near futur.
>>>>
>>>> What version of HttpClient are you using?
>>>>
>>>>> JM
>>>>>
>>>>> 2012/8/1, Oleg Kalnichevski <ol...@apache.org>:
>>>>>> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm getting the exception below when I'm trying to load
>>>>>>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
>>>>>>> ... It's not critical since I'm catching everything, but I'm
>>>>>>> wondering
>>>>>>> if this is normal or if this is something which need to be fixed...
>>>>>>>
>>>>>>> java.lang.NullPointerException
>>>>>>> at
>>>>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>>>>>> at
>>>>>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>>>>>> at
>>>>>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
>>>>>>> at
>>>>>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
>>>>>>> at
>>>>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
>>>>>>> at
>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>>>>>> at
>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>>>>>>> at
>>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>>>>>>> at
>>>>>>> org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> JM
>>>>>>>
>>>>>>
>>>>>> Jean-Marc
>>>>>>
>>>>>> I am not able to reproduce the problem just by executing HTTP GET to
>>>>>> the
>>>>>> URL in question. All looks perfectly normal.
>>>>>>
>>>>>> [DEBUG] BasicClientConnectionManager - Get connection for route
>>>>>> {}->http://www.expedia.ca
>>>>>> [DEBUG] DefaultClientConnectionOperator - Connecting to
>>>>>> www.expedia.ca:80
>>>>>> [DEBUG] RequestAddCookies - CookieSpec selected: best-match
>>>>>> [DEBUG] RequestAuthCache - Auth cache not set in the context
>>>>>> [DEBUG] RequestTargetAuthentication - Target auth state: UNCHALLENGED
>>>>>> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
>>>>>> [DEBUG] DefaultHttpClient - Attempt 1 to execute request
>>>>>> [DEBUG] DefaultClientConnection - Sending request:
>>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>>>> [DEBUG] headers - >>
>>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>>>> [DEBUG] headers - >> Host: www.expedia.ca
>>>>>> [DEBUG] headers - >> Connection: Keep-Alive
>>>>>> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
>>>>>> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
>>>>>> [DEBUG] headers - << HTTP/1.1 200 OK
>>>>>> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8
>>>>>> [DEBUG] headers - << Content-Language: en-CA
>>>>>> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND
>>>>>> COR
>>>>>> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
>>>>>> [DEBUG] headers - << RTSS: 1
>>>>>> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
>>>>>> [DEBUG] headers - << Transfer-Encoding: chunked
>>>>>> [DEBUG] headers - << Connection: keep-alive
>>>>>> [DEBUG] headers - << Connection: Transfer-Encoding
>>>>>> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
>>>>>> [DEBUG] headers - << Set-Cookie:
>>>>>> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
>>>>>> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
>>>>>> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
>>>>>> Path=/
>>>>>> [DEBUG] headers - << Set-Cookie:
>>>>>> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
>>>>>> Path=/
>>>>>> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/
>>>>>> [DEBUG] headers - << Set-Cookie: p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
>>>>>> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
>>>>>> 01-Aug-2017 15:54:40 GMT; Path=/
>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>>> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>>> MC1][value:
>>>>>> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path:
>>>>>> /][expiry:
>>>>>> Fri Aug 04 04:44:09 GMT+01:00 2017]".
>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>>> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry: null]".
>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>>> JSESSION][value:
>>>>>> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path:
>>>>>> /][expiry:
>>>>>> null]".
>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>>> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
>>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>>> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
>>>>>> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01 16:54:40
>>>>>> GMT+01:00 2017]".
>>>>>> [DEBUG] DefaultHttpClient - Connection can be kept alive indefinitely
>>>>>>
>>>>>> Can you put together a small test app that could be used to reproduce
>>>>>> the NPE?
>>>>>>
>>>>>> Oleg
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: NullPointerException in BestMatchSpec.formatCookies
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I'm using 1.7.0_05 from Sun.
I have enabled the RequestAddCookies debug logs and will see when I
will get the next NPE.
One the line 193 (I don't have the source) is version an int? Or an Integer?
I agree with you, cookie can't be null, else we will have got an NPE
at "if (!matchedCookies.isEmpty())"
I will restart my tool with the debug mode enabled and keep you posted.
2012/8/2, sebb <se...@gmail.com>:
> On 2 August 2012 19:03, Jean-Marc Spaggiari <je...@spaggiari.org>
> wrote:
>> I'm using version 4.2.1 binary distribution.
>
> In which case the NPE is very odd.
>
> The stack dump shows:
>
> java.lang.NullPointerException
> at
> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
> at
> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>
> Now the code in question is:
>
> BestMatchSpec.java:193: if (cookie.getVersion() < version) {
>
> AFAICT, this can only cause NPE if "cookie" is null.
>
> The cookie variable is derived from the list created in
> RequestAddCookies.java:
>
> List<Cookie> cookies = new
> ArrayList<Cookie>(cookieStore.getCookies());
> // Find cookies matching the given origin
> List<Cookie> matchedCookies = new ArrayList<Cookie>();
> Date now = new Date();
> for (Cookie cookie : cookies) {
> if (!cookie.isExpired(now)) { // <== cookie cannot be null here
> if (cookieSpec.match(cookie, cookieOrigin)) {
> if (this.log.isDebugEnabled()) {
> this.log.debug("Cookie " + cookie + " match "
> + cookieOrigin);
> }
> matchedCookies.add(cookie); // <== so cannot be null
> here
> }
> } else {
> if (this.log.isDebugEnabled()) {
> this.log.debug("Cookie " + cookie + " expired");
> }
> }
> }
> // Generate Cookie request headers
> if (!matchedCookies.isEmpty()) {
> List<Header> headers =
> cookieSpec.formatCookies(matchedCookies); // <== 196
> for (Header header : headers) {
> request.addHeader(header);
> }
> }
>
> I cannot see how a null cookie can be added to the list, nor how a
> list entry could be set to null later.
>
> It would be worth enabling Debug logging for the RequestAddCookies class.
>
> What JVM are you using?
>
>> 2012/8/2, sebb <se...@gmail.com>:
>>> On 1 August 2012 12:08, Jean-Marc Spaggiari <je...@spaggiari.org>
>>> wrote:
>>>> Hi Oleg,
>>>>
>>>> My application is reading many other URLs before this one. So maybe
>>>> it's a combination of multiple loads.
>>>>
>>>> I will try to dump more when I will get the issue again.
>>>>
>>>> In the meantime, here is how I'm initializing the HttpClient:
>>>>
>>>> // Creation of the HTTP Client
>>>> SchemeRegistry schemeRegistry = new SchemeRegistry();
>>>> schemeRegistry.register(new Scheme("http", 80,
>>>> PlainSocketFactory.getSocketFactory()));
>>>> schemeRegistry.register(new Scheme("https", 443,
>>>> SSLSocketFactory.getSocketFactory()));
>>>> PoolingClientConnectionManager cm = new
>>>> PoolingClientConnectionManager(schemeRegistry);
>>>> cm.setMaxTotal(200);
>>>> cm.setDefaultMaxPerRoute(20);
>>>>
>>>> HttpParams params = new BasicHttpParams();
>>>> if (useProxy)
>>>> params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new
>>>> HttpHost("proxy", 80));
>>>> client = new DefaultHttpClient(cm, params);
>>>> client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, new
>>>> Integer(30000));
>>>> client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
>>>> new Integer(30000));
>>>> client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY,
>>>> false);
>>>>
>>>> It's not the first time I'm getting this exception, so I might be able
>>>> to reproduce that in a near futur.
>>>
>>> What version of HttpClient are you using?
>>>
>>>> JM
>>>>
>>>> 2012/8/1, Oleg Kalnichevski <ol...@apache.org>:
>>>>> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm getting the exception below when I'm trying to load
>>>>>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
>>>>>> ... It's not critical since I'm catching everything, but I'm
>>>>>> wondering
>>>>>> if this is normal or if this is something which need to be fixed...
>>>>>>
>>>>>> java.lang.NullPointerException
>>>>>> at
>>>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>>>>> at
>>>>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>>>>> at
>>>>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
>>>>>> at
>>>>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
>>>>>> at
>>>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
>>>>>> at
>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>>>>> at
>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>>>>>> at
>>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>>>>>> at
>>>>>> org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>>>>>>
>>>>>>
>>>>>> --
>>>>>> JM
>>>>>>
>>>>>
>>>>> Jean-Marc
>>>>>
>>>>> I am not able to reproduce the problem just by executing HTTP GET to
>>>>> the
>>>>> URL in question. All looks perfectly normal.
>>>>>
>>>>> [DEBUG] BasicClientConnectionManager - Get connection for route
>>>>> {}->http://www.expedia.ca
>>>>> [DEBUG] DefaultClientConnectionOperator - Connecting to
>>>>> www.expedia.ca:80
>>>>> [DEBUG] RequestAddCookies - CookieSpec selected: best-match
>>>>> [DEBUG] RequestAuthCache - Auth cache not set in the context
>>>>> [DEBUG] RequestTargetAuthentication - Target auth state: UNCHALLENGED
>>>>> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
>>>>> [DEBUG] DefaultHttpClient - Attempt 1 to execute request
>>>>> [DEBUG] DefaultClientConnection - Sending request:
>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>>> [DEBUG] headers - >>
>>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>>> [DEBUG] headers - >> Host: www.expedia.ca
>>>>> [DEBUG] headers - >> Connection: Keep-Alive
>>>>> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
>>>>> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
>>>>> [DEBUG] headers - << HTTP/1.1 200 OK
>>>>> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8
>>>>> [DEBUG] headers - << Content-Language: en-CA
>>>>> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND
>>>>> COR
>>>>> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
>>>>> [DEBUG] headers - << RTSS: 1
>>>>> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
>>>>> [DEBUG] headers - << Transfer-Encoding: chunked
>>>>> [DEBUG] headers - << Connection: keep-alive
>>>>> [DEBUG] headers - << Connection: Transfer-Encoding
>>>>> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
>>>>> [DEBUG] headers - << Set-Cookie:
>>>>> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
>>>>> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
>>>>> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
>>>>> Path=/
>>>>> [DEBUG] headers - << Set-Cookie:
>>>>> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
>>>>> Path=/
>>>>> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/
>>>>> [DEBUG] headers - << Set-Cookie: p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
>>>>> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
>>>>> 01-Aug-2017 15:54:40 GMT; Path=/
>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>> MC1][value:
>>>>> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path:
>>>>> /][expiry:
>>>>> Fri Aug 04 04:44:09 GMT+01:00 2017]".
>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry: null]".
>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>> JSESSION][value:
>>>>> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path:
>>>>> /][expiry:
>>>>> null]".
>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
>>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>>> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
>>>>> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01 16:54:40
>>>>> GMT+01:00 2017]".
>>>>> [DEBUG] DefaultHttpClient - Connection can be kept alive indefinitely
>>>>>
>>>>> Can you put together a small test app that could be used to reproduce
>>>>> the NPE?
>>>>>
>>>>> Oleg
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: NullPointerException in BestMatchSpec.formatCookies
Posted by sebb <se...@gmail.com>.
On 2 August 2012 19:03, Jean-Marc Spaggiari <je...@spaggiari.org> wrote:
> I'm using version 4.2.1 binary distribution.
In which case the NPE is very odd.
The stack dump shows:
java.lang.NullPointerException
at org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
Now the code in question is:
BestMatchSpec.java:193: if (cookie.getVersion() < version) {
AFAICT, this can only cause NPE if "cookie" is null.
The cookie variable is derived from the list created in RequestAddCookies.java:
List<Cookie> cookies = new ArrayList<Cookie>(cookieStore.getCookies());
// Find cookies matching the given origin
List<Cookie> matchedCookies = new ArrayList<Cookie>();
Date now = new Date();
for (Cookie cookie : cookies) {
if (!cookie.isExpired(now)) { // <== cookie cannot be null here
if (cookieSpec.match(cookie, cookieOrigin)) {
if (this.log.isDebugEnabled()) {
this.log.debug("Cookie " + cookie + " match "
+ cookieOrigin);
}
matchedCookies.add(cookie); // <== so cannot be null here
}
} else {
if (this.log.isDebugEnabled()) {
this.log.debug("Cookie " + cookie + " expired");
}
}
}
// Generate Cookie request headers
if (!matchedCookies.isEmpty()) {
List<Header> headers =
cookieSpec.formatCookies(matchedCookies); // <== 196
for (Header header : headers) {
request.addHeader(header);
}
}
I cannot see how a null cookie can be added to the list, nor how a
list entry could be set to null later.
It would be worth enabling Debug logging for the RequestAddCookies class.
What JVM are you using?
> 2012/8/2, sebb <se...@gmail.com>:
>> On 1 August 2012 12:08, Jean-Marc Spaggiari <je...@spaggiari.org>
>> wrote:
>>> Hi Oleg,
>>>
>>> My application is reading many other URLs before this one. So maybe
>>> it's a combination of multiple loads.
>>>
>>> I will try to dump more when I will get the issue again.
>>>
>>> In the meantime, here is how I'm initializing the HttpClient:
>>>
>>> // Creation of the HTTP Client
>>> SchemeRegistry schemeRegistry = new SchemeRegistry();
>>> schemeRegistry.register(new Scheme("http", 80,
>>> PlainSocketFactory.getSocketFactory()));
>>> schemeRegistry.register(new Scheme("https", 443,
>>> SSLSocketFactory.getSocketFactory()));
>>> PoolingClientConnectionManager cm = new
>>> PoolingClientConnectionManager(schemeRegistry);
>>> cm.setMaxTotal(200);
>>> cm.setDefaultMaxPerRoute(20);
>>>
>>> HttpParams params = new BasicHttpParams();
>>> if (useProxy)
>>> params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new
>>> HttpHost("proxy", 80));
>>> client = new DefaultHttpClient(cm, params);
>>> client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, new
>>> Integer(30000));
>>> client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
>>> new Integer(30000));
>>> client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY, false);
>>>
>>> It's not the first time I'm getting this exception, so I might be able
>>> to reproduce that in a near futur.
>>
>> What version of HttpClient are you using?
>>
>>> JM
>>>
>>> 2012/8/1, Oleg Kalnichevski <ol...@apache.org>:
>>>> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
>>>>> Hi,
>>>>>
>>>>> I'm getting the exception below when I'm trying to load
>>>>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
>>>>> ... It's not critical since I'm catching everything, but I'm wondering
>>>>> if this is normal or if this is something which need to be fixed...
>>>>>
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>>>> at
>>>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>>>> at
>>>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
>>>>> at
>>>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
>>>>> at
>>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
>>>>> at
>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>>>> at
>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>>>>> at
>>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>>>>> at
>>>>> org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>>>>>
>>>>>
>>>>> --
>>>>> JM
>>>>>
>>>>
>>>> Jean-Marc
>>>>
>>>> I am not able to reproduce the problem just by executing HTTP GET to the
>>>> URL in question. All looks perfectly normal.
>>>>
>>>> [DEBUG] BasicClientConnectionManager - Get connection for route
>>>> {}->http://www.expedia.ca
>>>> [DEBUG] DefaultClientConnectionOperator - Connecting to
>>>> www.expedia.ca:80
>>>> [DEBUG] RequestAddCookies - CookieSpec selected: best-match
>>>> [DEBUG] RequestAuthCache - Auth cache not set in the context
>>>> [DEBUG] RequestTargetAuthentication - Target auth state: UNCHALLENGED
>>>> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
>>>> [DEBUG] DefaultHttpClient - Attempt 1 to execute request
>>>> [DEBUG] DefaultClientConnection - Sending request:
>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>> [DEBUG] headers - >>
>>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>>> [DEBUG] headers - >> Host: www.expedia.ca
>>>> [DEBUG] headers - >> Connection: Keep-Alive
>>>> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
>>>> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
>>>> [DEBUG] headers - << HTTP/1.1 200 OK
>>>> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8
>>>> [DEBUG] headers - << Content-Language: en-CA
>>>> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND COR
>>>> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
>>>> [DEBUG] headers - << RTSS: 1
>>>> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
>>>> [DEBUG] headers - << Transfer-Encoding: chunked
>>>> [DEBUG] headers - << Connection: keep-alive
>>>> [DEBUG] headers - << Connection: Transfer-Encoding
>>>> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
>>>> [DEBUG] headers - << Set-Cookie:
>>>> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
>>>> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
>>>> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
>>>> Path=/
>>>> [DEBUG] headers - << Set-Cookie:
>>>> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
>>>> Path=/
>>>> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/
>>>> [DEBUG] headers - << Set-Cookie: p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
>>>> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
>>>> 01-Aug-2017 15:54:40 GMT; Path=/
>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>> MC1][value:
>>>> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path:
>>>> /][expiry:
>>>> Fri Aug 04 04:44:09 GMT+01:00 2017]".
>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry: null]".
>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>> JSESSION][value:
>>>> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path:
>>>> /][expiry:
>>>> null]".
>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
>>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>>> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
>>>> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01 16:54:40
>>>> GMT+01:00 2017]".
>>>> [DEBUG] DefaultHttpClient - Connection can be kept alive indefinitely
>>>>
>>>> Can you put together a small test app that could be used to reproduce
>>>> the NPE?
>>>>
>>>> Oleg
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: NullPointerException in BestMatchSpec.formatCookies
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I'm using version 4.2.1 binary distribution.
2012/8/2, sebb <se...@gmail.com>:
> On 1 August 2012 12:08, Jean-Marc Spaggiari <je...@spaggiari.org>
> wrote:
>> Hi Oleg,
>>
>> My application is reading many other URLs before this one. So maybe
>> it's a combination of multiple loads.
>>
>> I will try to dump more when I will get the issue again.
>>
>> In the meantime, here is how I'm initializing the HttpClient:
>>
>> // Creation of the HTTP Client
>> SchemeRegistry schemeRegistry = new SchemeRegistry();
>> schemeRegistry.register(new Scheme("http", 80,
>> PlainSocketFactory.getSocketFactory()));
>> schemeRegistry.register(new Scheme("https", 443,
>> SSLSocketFactory.getSocketFactory()));
>> PoolingClientConnectionManager cm = new
>> PoolingClientConnectionManager(schemeRegistry);
>> cm.setMaxTotal(200);
>> cm.setDefaultMaxPerRoute(20);
>>
>> HttpParams params = new BasicHttpParams();
>> if (useProxy)
>> params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new
>> HttpHost("proxy", 80));
>> client = new DefaultHttpClient(cm, params);
>> client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, new
>> Integer(30000));
>> client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
>> new Integer(30000));
>> client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY, false);
>>
>> It's not the first time I'm getting this exception, so I might be able
>> to reproduce that in a near futur.
>
> What version of HttpClient are you using?
>
>> JM
>>
>> 2012/8/1, Oleg Kalnichevski <ol...@apache.org>:
>>> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
>>>> Hi,
>>>>
>>>> I'm getting the exception below when I'm trying to load
>>>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
>>>> ... It's not critical since I'm catching everything, but I'm wondering
>>>> if this is normal or if this is something which need to be fixed...
>>>>
>>>> java.lang.NullPointerException
>>>> at
>>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>>> at
>>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>>> at
>>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
>>>> at
>>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
>>>> at
>>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
>>>> at
>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>>> at
>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>>>> at
>>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>>>> at
>>>> org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>>>>
>>>>
>>>> --
>>>> JM
>>>>
>>>
>>> Jean-Marc
>>>
>>> I am not able to reproduce the problem just by executing HTTP GET to the
>>> URL in question. All looks perfectly normal.
>>>
>>> [DEBUG] BasicClientConnectionManager - Get connection for route
>>> {}->http://www.expedia.ca
>>> [DEBUG] DefaultClientConnectionOperator - Connecting to
>>> www.expedia.ca:80
>>> [DEBUG] RequestAddCookies - CookieSpec selected: best-match
>>> [DEBUG] RequestAuthCache - Auth cache not set in the context
>>> [DEBUG] RequestTargetAuthentication - Target auth state: UNCHALLENGED
>>> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
>>> [DEBUG] DefaultHttpClient - Attempt 1 to execute request
>>> [DEBUG] DefaultClientConnection - Sending request:
>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>> [DEBUG] headers - >>
>>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>>> [DEBUG] headers - >> Host: www.expedia.ca
>>> [DEBUG] headers - >> Connection: Keep-Alive
>>> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
>>> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
>>> [DEBUG] headers - << HTTP/1.1 200 OK
>>> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8
>>> [DEBUG] headers - << Content-Language: en-CA
>>> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND COR
>>> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
>>> [DEBUG] headers - << RTSS: 1
>>> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
>>> [DEBUG] headers - << Transfer-Encoding: chunked
>>> [DEBUG] headers - << Connection: keep-alive
>>> [DEBUG] headers - << Connection: Transfer-Encoding
>>> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
>>> [DEBUG] headers - << Set-Cookie:
>>> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
>>> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
>>> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
>>> Path=/
>>> [DEBUG] headers - << Set-Cookie:
>>> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
>>> Path=/
>>> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/
>>> [DEBUG] headers - << Set-Cookie: p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
>>> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
>>> 01-Aug-2017 15:54:40 GMT; Path=/
>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>> MC1][value:
>>> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path:
>>> /][expiry:
>>> Fri Aug 04 04:44:09 GMT+01:00 2017]".
>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry: null]".
>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>> JSESSION][value:
>>> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path:
>>> /][expiry:
>>> null]".
>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
>>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>>> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
>>> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01 16:54:40
>>> GMT+01:00 2017]".
>>> [DEBUG] DefaultHttpClient - Connection can be kept alive indefinitely
>>>
>>> Can you put together a small test app that could be used to reproduce
>>> the NPE?
>>>
>>> Oleg
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: NullPointerException in BestMatchSpec.formatCookies
Posted by sebb <se...@gmail.com>.
On 1 August 2012 12:08, Jean-Marc Spaggiari <je...@spaggiari.org> wrote:
> Hi Oleg,
>
> My application is reading many other URLs before this one. So maybe
> it's a combination of multiple loads.
>
> I will try to dump more when I will get the issue again.
>
> In the meantime, here is how I'm initializing the HttpClient:
>
> // Creation of the HTTP Client
> SchemeRegistry schemeRegistry = new SchemeRegistry();
> schemeRegistry.register(new Scheme("http", 80,
> PlainSocketFactory.getSocketFactory()));
> schemeRegistry.register(new Scheme("https", 443,
> SSLSocketFactory.getSocketFactory()));
> PoolingClientConnectionManager cm = new
> PoolingClientConnectionManager(schemeRegistry);
> cm.setMaxTotal(200);
> cm.setDefaultMaxPerRoute(20);
>
> HttpParams params = new BasicHttpParams();
> if (useProxy)
> params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new HttpHost("proxy", 80));
> client = new DefaultHttpClient(cm, params);
> client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, new
> Integer(30000));
> client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
> new Integer(30000));
> client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY, false);
>
> It's not the first time I'm getting this exception, so I might be able
> to reproduce that in a near futur.
What version of HttpClient are you using?
> JM
>
> 2012/8/1, Oleg Kalnichevski <ol...@apache.org>:
>> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
>>> Hi,
>>>
>>> I'm getting the exception below when I'm trying to load
>>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
>>> ... It's not critical since I'm catching everything, but I'm wondering
>>> if this is normal or if this is something which need to be fixed...
>>>
>>> java.lang.NullPointerException
>>> at
>>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>>> at
>>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>>> at
>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
>>> at
>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
>>> at
>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
>>> at
>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>>> at
>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>>> at
>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>>> at org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>>>
>>>
>>> --
>>> JM
>>>
>>
>> Jean-Marc
>>
>> I am not able to reproduce the problem just by executing HTTP GET to the
>> URL in question. All looks perfectly normal.
>>
>> [DEBUG] BasicClientConnectionManager - Get connection for route
>> {}->http://www.expedia.ca
>> [DEBUG] DefaultClientConnectionOperator - Connecting to
>> www.expedia.ca:80
>> [DEBUG] RequestAddCookies - CookieSpec selected: best-match
>> [DEBUG] RequestAuthCache - Auth cache not set in the context
>> [DEBUG] RequestTargetAuthentication - Target auth state: UNCHALLENGED
>> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
>> [DEBUG] DefaultHttpClient - Attempt 1 to execute request
>> [DEBUG] DefaultClientConnection - Sending request:
>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>> [DEBUG] headers - >>
>> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
>> [DEBUG] headers - >> Host: www.expedia.ca
>> [DEBUG] headers - >> Connection: Keep-Alive
>> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
>> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
>> [DEBUG] headers - << HTTP/1.1 200 OK
>> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8
>> [DEBUG] headers - << Content-Language: en-CA
>> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND COR
>> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
>> [DEBUG] headers - << RTSS: 1
>> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
>> [DEBUG] headers - << Transfer-Encoding: chunked
>> [DEBUG] headers - << Connection: keep-alive
>> [DEBUG] headers - << Connection: Transfer-Encoding
>> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
>> [DEBUG] headers - << Set-Cookie:
>> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
>> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
>> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
>> Path=/
>> [DEBUG] headers - << Set-Cookie:
>> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
>> Path=/
>> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/
>> [DEBUG] headers - << Set-Cookie: p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
>> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
>> 01-Aug-2017 15:54:40 GMT; Path=/
>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>> MC1][value:
>> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path: /][expiry:
>> Fri Aug 04 04:44:09 GMT+01:00 2017]".
>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry: null]".
>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>> JSESSION][value:
>> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path: /][expiry:
>> null]".
>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
>> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
>> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
>> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01 16:54:40
>> GMT+01:00 2017]".
>> [DEBUG] DefaultHttpClient - Connection can be kept alive indefinitely
>>
>> Can you put together a small test app that could be used to reproduce
>> the NPE?
>>
>> Oleg
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: NullPointerException in BestMatchSpec.formatCookies
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Oleg,
My application is reading many other URLs before this one. So maybe
it's a combination of multiple loads.
I will try to dump more when I will get the issue again.
In the meantime, here is how I'm initializing the HttpClient:
// Creation of the HTTP Client
SchemeRegistry schemeRegistry = new SchemeRegistry();
schemeRegistry.register(new Scheme("http", 80,
PlainSocketFactory.getSocketFactory()));
schemeRegistry.register(new Scheme("https", 443,
SSLSocketFactory.getSocketFactory()));
PoolingClientConnectionManager cm = new
PoolingClientConnectionManager(schemeRegistry);
cm.setMaxTotal(200);
cm.setDefaultMaxPerRoute(20);
HttpParams params = new BasicHttpParams();
if (useProxy)
params.setParameter(ConnRoutePNames.DEFAULT_PROXY, new HttpHost("proxy", 80));
client = new DefaultHttpClient(cm, params);
client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, new
Integer(30000));
client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,
new Integer(30000));
client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY, false);
It's not the first time I'm getting this exception, so I might be able
to reproduce that in a near futur.
JM
2012/8/1, Oleg Kalnichevski <ol...@apache.org>:
> On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
>> Hi,
>>
>> I'm getting the exception below when I'm trying to load
>> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
>> ... It's not critical since I'm catching everything, but I'm wondering
>> if this is normal or if this is something which need to be fixed...
>>
>> java.lang.NullPointerException
>> at
>> org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
>> at
>> org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
>> at
>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
>> at
>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
>> at
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
>> at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>> at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>> at
>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>> at org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>>
>>
>> --
>> JM
>>
>
> Jean-Marc
>
> I am not able to reproduce the problem just by executing HTTP GET to the
> URL in question. All looks perfectly normal.
>
> [DEBUG] BasicClientConnectionManager - Get connection for route
> {}->http://www.expedia.ca
> [DEBUG] DefaultClientConnectionOperator - Connecting to
> www.expedia.ca:80
> [DEBUG] RequestAddCookies - CookieSpec selected: best-match
> [DEBUG] RequestAuthCache - Auth cache not set in the context
> [DEBUG] RequestTargetAuthentication - Target auth state: UNCHALLENGED
> [DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
> [DEBUG] DefaultHttpClient - Attempt 1 to execute request
> [DEBUG] DefaultClientConnection - Sending request:
> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
> [DEBUG] headers - >>
> GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
> [DEBUG] headers - >> Host: www.expedia.ca
> [DEBUG] headers - >> Connection: Keep-Alive
> [DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
> [DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
> [DEBUG] headers - << HTTP/1.1 200 OK
> [DEBUG] headers - << Content-Type: text/html;charset=UTF-8
> [DEBUG] headers - << Content-Language: en-CA
> [DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND COR
> ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
> [DEBUG] headers - << RTSS: 1
> [DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
> [DEBUG] headers - << Transfer-Encoding: chunked
> [DEBUG] headers - << Connection: keep-alive
> [DEBUG] headers - << Connection: Transfer-Encoding
> [DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
> [DEBUG] headers - << Set-Cookie:
> MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
> Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
> [DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
> Path=/
> [DEBUG] headers - << Set-Cookie:
> JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
> Path=/
> [DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/
> [DEBUG] headers - << Set-Cookie: p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
> 0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
> 01-Aug-2017 15:54:40 GMT; Path=/
> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
> SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
> MC1][value:
> GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path: /][expiry:
> Fri Aug 04 04:44:09 GMT+01:00 2017]".
> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
> iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry: null]".
> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
> JSESSION][value:
> 13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path: /][expiry:
> null]".
> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
> s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
> [DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
> p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
> 0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01 16:54:40
> GMT+01:00 2017]".
> [DEBUG] DefaultHttpClient - Connection can be kept alive indefinitely
>
> Can you put together a small test app that could be used to reproduce
> the NPE?
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: NullPointerException in BestMatchSpec.formatCookies
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2012-07-31 at 17:37 -0400, Jean-Marc Spaggiari wrote:
> Hi,
>
> I'm getting the exception below when I'm trying to load
> http://www.expedia.ca/Sunset-District-Hotels.d163838.Travel-Guide-Hotels
> ... It's not critical since I'm catching everything, but I'm wondering
> if this is normal or if this is something which need to be fixed...
>
> java.lang.NullPointerException
> at org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:193)
> at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
> at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
> at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:515)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at org.spaggiari.distparser.CrawlerThread$1.run(CrawlerThread.java:267)
>
>
> --
> JM
>
Jean-Marc
I am not able to reproduce the problem just by executing HTTP GET to the
URL in question. All looks perfectly normal.
[DEBUG] BasicClientConnectionManager - Get connection for route
{}->http://www.expedia.ca
[DEBUG] DefaultClientConnectionOperator - Connecting to
www.expedia.ca:80
[DEBUG] RequestAddCookies - CookieSpec selected: best-match
[DEBUG] RequestAuthCache - Auth cache not set in the context
[DEBUG] RequestTargetAuthentication - Target auth state: UNCHALLENGED
[DEBUG] RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
[DEBUG] DefaultHttpClient - Attempt 1 to execute request
[DEBUG] DefaultClientConnection - Sending request:
GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
[DEBUG] headers - >>
GET /Sunset-District-Hotels.d163838.Travel-Guide-Hotels HTTP/1.1
[DEBUG] headers - >> Host: www.expedia.ca
[DEBUG] headers - >> Connection: Keep-Alive
[DEBUG] headers - >> User-Agent: Apache-HttpClient/4.2.1 (java 1.5)
[DEBUG] DefaultClientConnection - Receiving response: HTTP/1.1 200 OK
[DEBUG] headers - << HTTP/1.1 200 OK
[DEBUG] headers - << Content-Type: text/html;charset=UTF-8
[DEBUG] headers - << Content-Language: en-CA
[DEBUG] headers - << P3P: policyref="/w3c/p3p.xml", CP="CAO DSP IND COR
ADM CONo CUR CUSi DEV PSA PSD DELi OUR COM NAV PHY ONL PUR UNI"
[DEBUG] headers - << RTSS: 1
[DEBUG] headers - << Date: Wed, 01 Aug 2012 10:49:28 GMT
[DEBUG] headers - << Transfer-Encoding: chunked
[DEBUG] headers - << Connection: keep-alive
[DEBUG] headers - << Connection: Transfer-Encoding
[DEBUG] headers - << Set-Cookie: SSLB=1; path=/; domain=.expedia.ca
[DEBUG] headers - << Set-Cookie:
MC1=GUID=c55cfa1393ce4bfca1ac5b6532085717; Domain=.expedia.ca;
Expires=Fri, 04-Aug-2017 03:44:09 GMT; Path=/
[DEBUG] headers - << Set-Cookie: iEAPID=000000,; Domain=.expedia.ca;
Path=/
[DEBUG] headers - << Set-Cookie:
JSESSION=13201604-6f17-4f6e-8b03-570fd433ad4e; Domain=.expedia.ca;
Path=/
[DEBUG] headers - << Set-Cookie: s1=`0; Domain=.expedia.ca; Path=/
[DEBUG] headers - << Set-Cookie: p1=`tpid=v.1,4`linfo=v.4,|0|0|255|1|
0||||||||4105|0|0||0|0|0|-1|-1`63; Domain=.expedia.ca; Expires=Tue,
01-Aug-2017 15:54:40 GMT; Path=/
[DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
SSLB][value: 1][domain: .expedia.ca][path: /][expiry: null]".
[DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
MC1][value:
GUID=c55cfa1393ce4bfca1ac5b6532085717][domain: .expedia.ca][path: /][expiry: Fri Aug 04 04:44:09 GMT+01:00 2017]".
[DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
iEAPID][value: 000000,][domain: .expedia.ca][path: /][expiry: null]".
[DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
JSESSION][value:
13201604-6f17-4f6e-8b03-570fd433ad4e][domain: .expedia.ca][path: /][expiry: null]".
[DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
s1][value: `0][domain: .expedia.ca][path: /][expiry: null]".
[DEBUG] ResponseProcessCookies - Cookie accepted: "[version: 0][name:
p1][value: `tpid=v.1,4`linfo=v.4,|0|0|255|1|0||||||||4105|0|0||0|0|
0|-1|-1`63][domain: .expedia.ca][path: /][expiry: Tue Aug 01 16:54:40
GMT+01:00 2017]".
[DEBUG] DefaultHttpClient - Connection can be kept alive indefinitely
Can you put together a small test app that could be used to reproduce
the NPE?
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org