You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Rajapandian R <r....@verchaska.com> on 2006/02/10 16:18:45 UTC
Problem in downloading a page
Dear all
I am submitting some values to the page(http://www.goair.in/) using
PostMethod and i am expecting to get the next
page(http://www.goair.in/Booking/BERP_Availability.aspx), but i am not
able to get the page
The action of the form in page http://www.goair.in/ was
http://www.goair.in/besr_availability.aspx,
So i use the following code to initialise postmethod
PostMethod method2 = new
PostMethod("http://www.goair.in/Besr_availability.aspx");
I am expecting a redirect response, so that i can proceed and get the
expected page (http://www.goair.in/Booking/BERP_Availability.aspx) ,
but i am not getting any redirect response,instead of that i am getting
a 200 OK response with content of length zero.
Please help to solve this problem.
regards
rajapandian
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem in downloading a page
Posted by Rajapandian R <r....@verchaska.com>.
Dear Roland
In the previous mail i have sent u the code in which i encounter some
problems .I execute the same after some time,I got the result.
So i tried it with some intervals, I get the expected result page some
times and i am not getting it some times.
I am not getting the required page consistantly.
What may be the problem,how to rectify this,give me some suggestions.
Thanks
regards
rajapandian
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem in downloading a page
Posted by Roland Weber <ht...@dubioso.net>.
Hi Rajapandian,
remove the Accept-Encoding headers from your request. Otherwise,
your application must be prepared to decompress responses. (I'll
remember this next time I write a checklist. :-) That is not
the reason for your problems though, the server does not send
a compressed response.
> *Response headers of request 2 (Post Request)*
> Content-Length: 0
> Cache-Control: private
> Response Status : HTTP/1.1 200 OK
That is very suspicious, and extremely hard to debug. There is
no transmission problem: the server simply sends you an empty
response. No error message to work with. I have seen such a
behavior when stressing a local development environment with
frequent changes, but not with a public web server.
Any ideas, anybody?
One last idea: let your application save the document received
with the initial GET request. Use different filenames each time,
for example by including a timestamp in the filename. Compare
a document from a successful attempt with one from a failed
attempt, to see whether there is a difference. For example, the
document in the failed attempt could read "you have to wait
30 seconds before sending your next request". Or there might
be a difference in the form parameters.
cheers,
Roland
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem in downloading a page
Posted by Rajapandian R <r....@verchaska.com>.
Dear Roland
I tried by giving intervals between the request and enablethe
expect-continue handshake for the POST request(s).but i am facing the
same problem
I have pasted the response headers when i am getting the result and not
getting the result,please have a look.
*Request/Response headers when i am not getting the result :
**Request headers of request 2 (Post Request)*
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
Cache-Control: no-cache
*Response headers of request 2 (Post Request)*
Content-Length: 0
Cache-Control: private
Response Status : HTTP/1.1 200 OK
*Request headers of request 3
*Contains no Content-Type header
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
*Response headers of request 3*
Content-Length: 1930
Content-Type: text/html; charset=utf-8
Cache-Control: private
Response Status : HTTP/1.1 200 OK
-------------------------------------------------------------------------------------------------------------------------------------------
*Response headers when i am getting the result :
**Request headers of request 2 (Post Request)*
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
Cache-Control: no-cache
*Response headers of request 2*
Content-Length: 40482
Content-Type: text/html; charset=utf-8
Cache-Control: private
Response Status : HTTP/1.1 200 OK
*Request headers of request 3
*Contains no Content-Type header
Accept-Encoding: gzip,deflate
Connection: Keep-Alive
*Response headers of request 3*
Content-Length: 41616
Content-Type: text/html; charset=utf-8
Cache-Control: private
Response Status : HTTP/1.1 200 OK
regards
rajapandian
Re: Problem in downloading a page
Posted by Roland Weber <ht...@dubioso.net>.
Hello Rajapandian,
Rajapandian R wrote:
> i use ie http analyser and get the required informations and i set all
> the request headers and i give the required form values .
> i checked the cookie values,every thing is right, the response is 200
> ok,but the content length is zero.i have pasted the code below.
> please have a look and suggest me some ideas.
> homepage.setRequestHeader("Host","www.goair.in");
> homepage.setRequestHeader("Connection","Keep-Alive");
These headers are controlled by HttpClient. DON'T SET THEM!
In particular not the Host header, the other one is less problematic.
> searchresult.setRequestHeader("Request-Line","POST/besr_availability.aspx
> HTTP/1.1");
You can't be serious about this one. If you used an HTTP analyzer to
compare the requests sent by HttpClient and the browser, didn't you
notice the difference? Please compare the requests again, and make
sure that the one from HttpClient doesn't send any more garbage.
Also, have a quick look into RFC 2616 to learn the difference between
a header and a request line: http://www.ietf.org/rfc/rfc2616.txt
There are also nice PDF versions of this RFC if you search for them.
> searchresult.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
> searchresult.setRequestHeader("Host","goair.in");
> searchresult.setRequestHeader("Connection","Keep-Alive");
These headers are controlled by HttpClient. DON'T SET THEM!
In particular not the Host and Content-Type headers. The
other one is less problematic.
> [snipped more code]
And so on. I'm not here to debug your application. Please tell me about
status codes and headers you get or do not get. I have no intention
whatsoever of setting up an environment for running your code to get
that data myself.
In a second mail, you wrote:
> So i tried it with some intervals, I get the expected result page
> some times and i am not getting it some times.
> I am not getting the required page consistantly.
What's the value of the Content-Length, Content-Type, Transfer-Encoding
headers if you do or do not get the result page? You can try to put some
delay into your application between the requests. A user with a browser
doesn't click with response times in the milliseconds range. Maybe the
site has some protection against DoS attacks. You can also try to enable
the expect-continue handshake for the POST request(s).
http://jakarta.apache.org/commons/httpclient/performance.html#Expect-continue%20handshake
hope that helps,
Roland
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem in downloading a page
Posted by Rajapandian R <r....@verchaska.com>.
Dear Roland
> 1. did you execute a GET before the POST to establish a session?
>
yes the initial request was Get request
> 2. are session cookies obtained with the GET sent back as expected?
> Play around with cookie policies and "single-cookie-header", as
> mentioned in the cookie guide.
>
I didnt handled any cookie policies, but checked the Cookie request
header of the second and third request, it was ok.
> 3. are you sure you send all required parameters with the POST?
> In the same format as indicated by the <form> in the web page?
> (multipart vs. URL-encoded)
>
yes i am sure, since i use ie http analyser and get the required parameters
> 4. try sending a User-Agent header with an IE or Mozilla value
>
> 5. try sending a Referer-Header (spelling intentional)
>
> 6. get a network monitor, grab the request sent by the browser,
> then compare the one generated by HttpClient line-by-line
>
> 7. if you have problems creating a request identical to the one
> generated by the browser, post a *detailed* description of
> what you want to send and what is generated by HttpClient
>
i use ie http analyser and get the required informations and i set all
the request headers and i give the required form values .
i checked the cookie values,every thing is right, the response is 200
ok,but the content length is zero.i have pasted the code below.
please have a look and suggest me some ideas.
--------------------------------------------------------------------------------------------------------------
import java.awt.HeadlessException;
import java.io.IOException;
import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.TraceMethod;
public class GoAirAvailability {
public static void main(String[] args) throws IOException,
HeadlessException {
HttpClient client = new HttpClient();
try{
GetMethod homepage = new GetMethod("http://www.goair.in/");
homepage.setRequestHeader("Request-Line","GET /HTTP/1.1");
homepage.setRequestHeader("Accept","image/gif,image/x-xbitmap,
image/jpeg,
image/pjpeg,application/x-shockwave-flash,application/vnd.ms-excel,application/vnd.ms-powerpoint,
application/msword,*/*");
homepage.setRequestHeader("Accept-Language","en-us");
homepage.setRequestHeader("Accept-Encoding","gzip,deflate");
homepage.setRequestHeader("User-Agent","Mozilla/4.0(compatible; MSIE
6.0; Windows NT 5.1; SV1)");
homepage.setRequestHeader("Host","www.goair.in");
homepage.setRequestHeader("Connection","Keep-Alive");
int statusCode0 = client.executeMethod(homepage);
System.out.println("Search
page::flykingfisher.biz"+homepage.getStatusLine());
System.out.println("Response headers of request1 (home
page)-----------------------------------------------");
Header[] homepageres = homepage.getResponseHeaders();
for (int i = 0; i < homepageres.length; i++) {
System.out.println(homepageres[i].toExternalForm());
}
System.out.println("--------------------------------------------------------------------------------------");
PostMethod searchresult = new
PostMethod("http://www.goair.in/besr_availability.aspx");
searchresult.setRequestHeader("Request-Line","POST/besr_availability.aspx
HTTP/1.1");
searchresult.setRequestHeader("Accept","image/gif,image/x-xbitmap,
image/jpeg,
image/pjpeg,application/x-shockwave-flash,application/vnd.ms-excel,application/vnd.ms-powerpoint,
application/msword,*/*");
searchresult.setRequestHeader("Referer","http://goair.in/");
searchresult.setRequestHeader("Accept-Language","en-us");
searchresult.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
searchresult.setRequestHeader("Accept-Encoding","gzip,deflate");
searchresult.setRequestHeader("User-Agent","Mozilla/4.0(compatible;
MSIE 6.0; Windows NT 5.1; SV1)");
searchresult.setRequestHeader("Host","goair.in");
//searchresult.setRequestHeader("Content-Length","551");
searchresult.setRequestHeader("Connection","Keep-Alive");
searchresult.setRequestHeader("Cache-Control","no-cache");
//searchresult.setRequestHeader("Cookie","ASP.NET_SessionId=wsdgmsbpbb5k3prikmg50c45");
NameValuePair nvp1 = new
NameValuePair("cboNationality","IN");
NameValuePair nvp2 = new NameValuePair("hidRemote","");
NameValuePair nvp3 = new
NameValuePair("__EVENTTARGET","BE_SearchBar11:lnkSearch");
NameValuePair nvp4 = new NameValuePair("rbRound","one");
NameValuePair nvp5 = new NameValuePair("cboChild","0");
NameValuePair nvp6 = new
NameValuePair("__EVENTARGUMENT","");
NameValuePair nvp7 = new NameValuePair("cboInfant","0");
NameValuePair nvp8 = new NameValuePair("hidDestination","");
NameValuePair nvp9 = new
NameValuePair("__VIEWSTATE","dDwyODczMTEwMjk7dDw7bDxpPDE+Oz47bDx0PDtsPGk8Mz47PjtsPHQ8O2w8aTwwPjtpPDE+Oz47bDx0PHA8O3A8bDxvbmNsaWNrOz47bDxyZXR1cm4gdmFsaWRhdGVfU2VhcmNoKCk7Pj4+Ozs+O3Q8cDw7cDxsPG9uY2xpY2s7PjtsPHJldHVybiB2YWxpZGF0ZV9TZWFyY2goKTs+Pj47Oz47Pj47Pj47Pj47PjiegUxbAvWlc5Hrx0/8gVkHebp9");
NameValuePair nvp10 = new NameValuePair("cboClass","3");
NameValuePair nvp11 = new NameValuePair("hidRound","one");
NameValuePair nvp12 = new
NameValuePair("cbodepdayyear","3,2006");
NameValuePair nvp13 = new NameValuePair("cboAdult","1");
NameValuePair nvp14 = new NameValuePair("hidRetDate","");
NameValuePair nvp15 = new
NameValuePair("cboOrigin","GOI7");
NameValuePair nvp16 = new NameValuePair("cbodepday","11");
NameValuePair nvp17 = new NameValuePair("hidOrigin","");
NameValuePair nvp18 = new
NameValuePair("cboDestination","BOM1");
//NameValuePair nvp19 = new
NameValuePair("cbomonthyearret","3,2006");
// NameValuePair nvp20 = new NameValuePair("cboretday","10");
searchresult.setRequestBody(new
NameValuePair[]{nvp1,nvp2,nvp3,nvp4,nvp5,nvp6,nvp7,nvp8,nvp9,nvp10,
nvp11,nvp12,nvp13,nvp14,nvp15,nvp16,nvp17,nvp18});//,nvp19,nvp20
/*
searchresult.setRequestBody("__EVENTTARGET=BE_SearchBar11%3AlnkSearch&__EVENTARGUMENT=&__VIEWSTATE=dDwyODczMTEwMjk7dDw7bDxpPDE%2BOz47bDx0PDtsPGk8Mz47PjtsPHQ8O2w8aTwwPjtpPDE%2BOz47bDx0PHA8O3A8bDxvbmNsaWNrOz47bDxyZXR1cm4gdmFsaWRhdGVfU2VhcmNoKCk7Pj4%2BOzs%2BO3Q8cDw7cDxsPG9uY2xpY2s7PjtsPHJldHVybiB2YWxpZGF0ZV9TZWFyY2goKTs%2BPj47Oz47Pj47Pj47Pj47Pkx%2B1JMmXFaM8Kb85kapAZSVJvDx&"
+
"rbRound=one&hidRound=one&hidRemote=&cboOrigin=GOI7&hidOrigin=&cboDestination=BOM1&hidDestination=&"
+
"cbodepday=11&cbodepdayyear=3%2C2006&hidRetDate=&cboAdult=1&cboInfant=0&cboChild=0&cboClass=3&"
+
"cboNationality=IN");
*/
int status1 = client.executeMethod(searchresult);
System.out.println(searchresult.getResponseBodyAsString());
System.out.println("Request headers of request
2-----------------------------------------------");
Header[] searchresultreq =
searchresult.getRequestHeaders();
for (int i = 0; i < searchresultreq.length; i++)
{
System.out.println(searchresultreq[i].toExternalForm());
}
System.out.println("--------------------------------------------------------------------------------------");
System.out.println(searchresult.getRequestHeader("Cookie"));
System.out.println("Search
page::flykingfisher.biz"+homepage.getStatusLine());
System.out.println("Response headers of request
2-----------------------------------------------");
Header[] searchresultres =
searchresult.getResponseHeaders();
for (int i = 0; i < searchresultres.length; i++)
{
System.out.println(searchresultres[i].toExternalForm());
}
System.out.println("--------------------------------------------------------------------------------------");
GetMethod searchresultpage = new
GetMethod("http://www.goair.in/Booking/BERP_Availability.aspx");
searchresultpage.setRequestHeader("Request-Line","GET/Booking/BERP_Availability.aspx
HTTP/1.1");
searchresultpage.setRequestHeader("Accept","image/gif,image/x-xbitmap,
image/jpeg,
image/pjpeg,application/x-shockwave-flash,application/vnd.ms-excel,application/vnd.ms-powerpoint,
application/msword,*/*");
searchresultpage.setRequestHeader("Accept-Language","en-us");
searchresultpage.setRequestHeader("Accept-Encoding","gzip,deflate");
searchresultpage.setRequestHeader("User-Agent","Mozilla/4.0(compatible;
MSIE 6.0; Windows NT 5.1; SV1)");
searchresultpage.setRequestHeader("Host","goair.in");
searchresultpage.setRequestHeader("Connection","Keep-Alive");
//searchresultpage.setRequestHeader("Cookie","ASP.NET_SessionId=wsdgmsbpbb5k3prikmg50c45");
int finalstatus = client.executeMethod(searchresultpage);
System.out.println(searchresultpage.getRequestHeader("Cookie"));
System.out.println(searchresultpage.getResponseBodyAsString());
System.out.println("Response headers of request 2
-----------------------------------------------");
Header[] searchresultpageres =
searchresultpage.getResponseHeaders();
for (int i = 0; i < searchresultpageres.length;i++) {
System.out.println(searchresultpageres[i].toExternalForm());
}
System.out.println("--------------------------------------------------------------------------------------");
}
catch(IOException e)
{
System.out.println(e);
}
}
}
regards
rajapandian.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Problem in downloading a page
Posted by Roland Weber <ht...@dubioso.net>.
Hi Rajapandian,
please go through the usual checklist:
1. did you execute a GET before the POST to establish a session?
2. are session cookies obtained with the GET sent back as expected?
Play around with cookie policies and "single-cookie-header", as
mentioned in the cookie guide.
3. are you sure you send all required parameters with the POST?
In the same format as indicated by the <form> in the web page?
(multipart vs. URL-encoded)
4. try sending a User-Agent header with an IE or Mozilla value
5. try sending a Referer-Header (spelling intentional)
6. get a network monitor, grab the request sent by the browser,
then compare the one generated by HttpClient line-by-line
7. if you have problems creating a request identical to the one
generated by the browser, post a *detailed* description of
what you want to send and what is generated by HttpClient
hope that helps,
Roland
Rajapandian R wrote:
> Dear all
>
> I am submitting some values to the page(http://www.goair.in/) using
> PostMethod and i am expecting to get the next
> page(http://www.goair.in/Booking/BERP_Availability.aspx), but i am not
> able to get the page
>
>
> The action of the form in page http://www.goair.in/ was
> http://www.goair.in/besr_availability.aspx,
>
> So i use the following code to initialise postmethod
> PostMethod method2 = new
> PostMethod("http://www.goair.in/Besr_availability.aspx");
>
> I am expecting a redirect response, so that i can proceed and get the
> expected page (http://www.goair.in/Booking/BERP_Availability.aspx) ,
> but i am not getting any redirect response,instead of that i am getting
> a 200 OK response with content of length zero.
>
> Please help to solve this problem.
>
> regards
> rajapandian
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org