You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Robert Stone <rt...@gmail.com> on 2010/02/15 16:36:53 UTC
Re: Logging into a website issues with 4.0.1
*Thanks to Jeff* for getting me on my way to figuring this issue out. I took
two wireshark logs to see what the differences are between the automated
login and the manual login just as he suggested. The text of the wireshark
logs are as follows.
The Manual Login:
-----------------------------------------------------------------------------------------------------------------------------
POST /div/login HTTP/1.1
Host: my.website.com:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)
AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5
Referer: http://my.website.com:8080/div/login
Content-Length: 38
Cache-Control: max-age=0
Origin: http://my.website.com:8080
Content-Type: application/x-www-form-urlencoded
Accept:
application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Cookie: JSESSIONID=AF810830BBC3D8869D70178E73239217
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
txtUserName=myUserName&txtPassword=myPassWordHTTP/1.1 302 Moved Temporarily
Server: Apache-Coyote/1.1
Location: http://my.website.com:8080/div/
Content-Length: 0
Date: Fri, 12 Feb 2010 01:52:30 GMT
------------------------------------------------------------------------------------------------------------------------------------
The Automated Login
-----------------------------------------------------------------------------------------------------------------------------------
POST /div/login.jsp?txtUserName=myUserName&txtPassword=myPassWord HTTP/1.1
Content-Length: 0
Host: my.website.com:8080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.0.1 (java 1.5)
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=9563FDBAA69961D49B607C63383C5F7A; Path=/div
Cache-Control: no-cache, must-revalidate
Expires: 0
Date: Fri, 12 Feb 2010 17:09:24 GMT
Content-Type: text/xml;charset=UTF-8
Content-Length: 3648
----------------------------------------------------------------------------------------------------
As one can clearly see the Manual Login contains far much more information
in the Packet. I have no idea how to include this information in the request
that I am sending from the http-client class I made. I looked for methods
with auto complete that would allow me to add this information however I
found nothing specific. Can anyone get me started on the road to adding this
information to my automated login's packet.
Originally I attached my source code in a zip file to make the email more
readable however this mail list doesn't seem to allow attachments so here is
the source code in text format.
Sorry in advance if it is ugly or hard to read.
First the Java class
-----------------------------------------------------------------------------------'
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.List;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.utils.URIUtils;
import org.apache.http.client.utils.URLEncodedUtils;
import org.apache.http.cookie.Cookie;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HTTP;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;
/**
* WebUtils This class contains basic Web Utilities used for logging into
and manipulating data
* on a requested website.
*
* @author ME
*/
public class WebUtils {
/**
* getPageSource This Method will return the source code of the
requested page when given the
* following parameters
*
* @return pageSource The source code of the requested page
*/
public String getPageSource(){
String pageSource = "";
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
List<NameValuePair> qparams = new
ArrayList<NameValuePair>();
qparams.add(new BasicNameValuePair("txtUserName",
"myusername"));
qparams.add(new BasicNameValuePair("txtPassword",
"mypassword"));
URI uri;
try {
uri = URIUtils.createURI("http", "my.website.com",
8080, "/div/login",
URLEncodedUtils.format(qparams, "UTF-8"),
null);
HttpPost httppost = new HttpPost(uri);
HttpResponse response = httpclient.execute(httppost,
localContext);
HttpEntity entity = response.getEntity();
List<Cookie> cookies =
httpclient.getCookieStore().getCookies();
if (cookies.isEmpty()) {
System.out.println("None");
} else {
for (int i = 0; i < cookies.size(); i++) {
System.out.println(i +" - " +
cookies.get(i).toString());
}
}
if (entity != null) {
long len = entity.getContentLength();
if (len != -1) {
pageSource = EntityUtils.toString(entity);
} else {
pageSource = "";
}
}
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return pageSource;
}
}
The Form
------------------------------------------------------------------------------------------------------
<div class="loginPanel">
<div class="container">
<div class="title">Login</div>
<form id="loginForm" class="loginForm" method="POST"
action="/div/login">
<font class="portlet-msg-error"
style="font-weight: bold; font-size: 10px; color:#FF0000;
text-align: center;"></font>
<table>
<tr>
<td class="label">USERNAME:</td>
<td><input
value=""
class="edit"
id="txtUsernameLogin"
name="txtUserName"
type="text"/></td>
</tr>
<tr>
<td class="label">PASSWORD:</td>
<td><input
value=""
class="edit"
name="txtPassword"
type="password"/></td>
</tr>
<tr>
<td colspan="2" align="center">
<input type="checkbox" name="txtRemember"
value="true" align="middle"/>
Remember me on this computer
</td>
</tr>
<tr>
<td></td>
<td class=""><input type="submit"
class="submit"
value="Login"
alt="Login"/></td>
</tr>
</table>
</form>
</div>
</div>
Any help will be greatly appreciated.
Thanks,
Robert
Re: Logging into a website issues with 4.0.1
Posted by Robert Stone <rt...@gmail.com>.
I checked out the link and the method referenced below and I still have had
no success logging into my site or even the site in the example. I did
notice that the site in the example has changed so I modified the example to
meet the changes in the site. However, I still can not login. Can you please
tell me what I am doing wrong.
Below you will find my login method code that I am using. This is almost
straight from the example at this link
http://svn.apache.org/repos/asf/httpcomponents/httpclient/branches/4.0.x/httpclient/src/examples/org/apache/http/examples/client/ClientFormLogin.java
<http://svn.apache.org/repos/asf/httpcomponents/httpclient/branches/4.0.x/httpclient/src/examples/org/apache/http/examples/client/ClientFormLogin.java>
public String login() throws Exception {
String pageSource = "";
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet("https://portal.sun.com/portal/dt");
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println("Login form get: " + response.getStatusLine());
if (entity != null) {
entity.consumeContent();
}
System.out.println("Initial set of cookies:");
List<Cookie> cookies = httpclient.getCookieStore().getCookies();
if (cookies.isEmpty()) {
System.out.println("None");
} else {
for (int i = 0; i < cookies.size(); i++) {
System.out.println("- " + cookies.get(i).toString());
}
}
HttpPost httpost = new HttpPost("
https://identity.sun.com/amserver/UI/Login?" +
"org=self_registered_users&" +
"program=sdn&" +
"goto=
http://reg.sun.com/updateaccount%3Fprogram%3Dsdn%26goto%3Dhttp://developers.sun.com
");
List <NameValuePair> nvps = new ArrayList <NameValuePair>();
nvps.add(new BasicNameValuePair("IDToken1", "yourUserNameHere"));
nvps.add(new BasicNameValuePair("IDToken2", "yourPasswordHere"));
httpost.setEntity(new UrlEncodedFormEntity(nvps, HTTP.UTF_8));
response = httpclient.execute(httpost);
entity = response.getEntity();
System.out.println("Login form get: " + response.getStatusLine());
if (entity != null) {
long len = entity.getContentLength();
System.out.println("len is=> " + len);
if (len != 0) {
System.out.println("Getting ready to set the page source var");
pageSource = EntityUtils.toString(entity);
System.out.println("After setting the page source var");
entity.consumeContent();
} else {
pageSource = "I suck I can't login!";
}
}
System.out.println("Post logon cookies:");
cookies = httpclient.getCookieStore().getCookies();
if (cookies.isEmpty()) {
System.out.println("None");
} else {
for (int i = 0; i < cookies.size(); i++) {
System.out.println("- " + cookies.get(i).toString());
}
}
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
return pageSource;
}
}
Thanks in advance,
Robert
//////////////////////////////////////////////////////////////////////////////////////////////////////////////
On Mon, Feb 15, 2010 at 2:46 PM, Ken Krugler <kk...@transpac.com>wrote:
>
> On Feb 15, 2010, at 10:07am, Robert Stone wrote:
>
> Ken,
>>
>> They are in there. The problem is that when I send them from the script it
>> is putting them in a different place then the manual. If you look at the
>> wireshark log again you will see they are actually at the top for some
>> reason not at the bottom like in the manual.
>>
>
> Those are URL parameters, not POST parameters.
>
> See
> http://svn.apache.org/repos/asf/httpcomponents/httpclient/branches/4.0.x/httpclient/src/examples/org/apache/http/examples/client/ClientFormLogin.java
>
> and specifically the httppost.setEntity() call.
>
> -- Ken
>
>
> ------------------------------------------------------------------------------------------------------------------------------------
>> The Automated Login
>>
>> -----------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> POST /div/login.jsp?txtUserName=myUserName&txtPassword=myPassWord HTTP/1.1
>> <===== Here is where they are in the automated
>>
>> I don't know why this is because I think I am sending them properly. Here
>> is
>> the section from my code.
>> List<NameValuePair> qparams = new
>> ArrayList<NameValuePair>();
>> qparams.add(new BasicNameValuePair("txtUserName",
>> "myusername")); <== Here are the user name and password
>> qparams.add(new BasicNameValuePair("txtPassword",
>> "mypassword"));
>>
>> URI uri;
>> try {
>> uri = URIUtils.createURI("http", "my.website.com",
>> 8080, "/div/login",
>> URLEncodedUtils.format(qparams, "UTF-8"),
>> null); <== Here is where I ad the params to the post
>>
>> HttpPost httppost = new HttpPost(uri); <===== and
>> here is where I am sending the post.
>>
>> Any Ideas what is going wrong?
>>
>> Thanks,
>>
>> Robert
>>
>>
>> On Mon, Feb 15, 2010 at 10:36 AM, Ken Krugler
>> <kk...@transpac.com>wrote:
>>
>> In the automated login, I don't see the POST parameters:
>>>
>>> txtUserName=myUserName&txtPassword=myPassWord
>>>
>>> -- Ken
>>>
>>> On Feb 15, 2010, at 7:36am, Robert Stone wrote:
>>>
>>> *Thanks to Jeff* for getting me on my way to figuring this issue out. I
>>>
>>>> took
>>>>
>>>> two wireshark logs to see what the differences are between the automated
>>>> login and the manual login just as he suggested. The text of the
>>>> wireshark
>>>> logs are as follows.
>>>>
>>>> The Manual Login:
>>>>
>>>>
>>>> -----------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> POST /div/login HTTP/1.1
>>>> Host: my.website.com:8080
>>>> Connection: keep-alive
>>>> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)
>>>> AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5
>>>> Referer: http://my.website.com:8080/div/login
>>>> Content-Length: 38
>>>> Cache-Control: max-age=0
>>>> Origin: http://my.website.com:8080
>>>> Content-Type: application/x-www-form-urlencoded
>>>> Accept:
>>>>
>>>>
>>>> application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
>>>> Accept-Encoding: gzip,deflate,sdch
>>>> Cookie: JSESSIONID=AF810830BBC3D8869D70178E73239217
>>>> Accept-Language: en-US,en;q=0.8
>>>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
>>>>
>>>> txtUserName=myUserName&txtPassword=myPassWordHTTP/1.1 302 Moved
>>>> Temporarily
>>>> Server: Apache-Coyote/1.1
>>>> Location: http://my.website.com:8080/div/
>>>> Content-Length: 0
>>>> Date: Fri, 12 Feb 2010 01:52:30 GMT
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------------------------------------------------------------
>>>> The Automated Login
>>>>
>>>>
>>>> -----------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> POST /div/login.jsp?txtUserName=myUserName&txtPassword=myPassWord
>>>> HTTP/1.1
>>>> Content-Length: 0
>>>> Host: my.website.com:8080
>>>> Connection: Keep-Alive
>>>> User-Agent: Apache-HttpClient/4.0.1 (java 1.5)
>>>>
>>>> HTTP/1.1 200 OK
>>>> Server: Apache-Coyote/1.1
>>>> Set-Cookie: JSESSIONID=9563FDBAA69961D49B607C63383C5F7A; Path=/div
>>>> Cache-Control: no-cache, must-revalidate
>>>> Expires: 0
>>>> Date: Fri, 12 Feb 2010 17:09:24 GMT
>>>> Content-Type: text/xml;charset=UTF-8
>>>> Content-Length: 3648
>>>>
>>>>
>>>>
>>>> ----------------------------------------------------------------------------------------------------
>>>>
>>>> As one can clearly see the Manual Login contains far much more
>>>> information
>>>> in the Packet. I have no idea how to include this information in the
>>>> request
>>>> that I am sending from the http-client class I made. I looked for
>>>> methods
>>>> with auto complete that would allow me to add this information however I
>>>> found nothing specific. Can anyone get me started on the road to adding
>>>> this
>>>> information to my automated login's packet.
>>>>
>>>> Originally I attached my source code in a zip file to make the email
>>>> more
>>>> readable however this mail list doesn't seem to allow attachments so
>>>> here
>>>> is
>>>> the source code in text format.
>>>>
>>>> Sorry in advance if it is ugly or hard to read.
>>>>
>>>> First the Java class
>>>>
>>>>
>>>> -----------------------------------------------------------------------------------'
>>>>
>>>> import java.io.IOException;
>>>> import java.net.URI;
>>>> import java.net.URISyntaxException;
>>>> import java.util.ArrayList;
>>>> import java.util.List;
>>>> import org.apache.http.HttpEntity;
>>>> import org.apache.http.HttpResponse;
>>>> import org.apache.http.NameValuePair;
>>>> import org.apache.http.client.ClientProtocolException;
>>>> import org.apache.http.client.entity.UrlEncodedFormEntity;
>>>> import org.apache.http.client.methods.HttpGet;
>>>> import org.apache.http.client.methods.HttpPost;
>>>> import org.apache.http.client.utils.URIUtils;
>>>> import org.apache.http.client.utils.URLEncodedUtils;
>>>> import org.apache.http.cookie.Cookie;
>>>> import org.apache.http.impl.client.DefaultHttpClient;
>>>> import org.apache.http.message.BasicNameValuePair;
>>>> import org.apache.http.protocol.BasicHttpContext;
>>>> import org.apache.http.protocol.HTTP;
>>>> import org.apache.http.protocol.HttpContext;
>>>> import org.apache.http.util.EntityUtils;
>>>>
>>>> /**
>>>> * WebUtils This class contains basic Web Utilities used for logging into
>>>> and manipulating data
>>>> * on a requested website.
>>>> *
>>>> * @author ME
>>>> */
>>>> public class WebUtils {
>>>>
>>>>
>>>> /**
>>>> * getPageSource This Method will return the source code of the
>>>> requested page when given the
>>>> * following parameters
>>>> *
>>>> * @return pageSource The source code of the requested page
>>>> */
>>>> public String getPageSource(){
>>>>
>>>> String pageSource = "";
>>>>
>>>> DefaultHttpClient httpclient = new DefaultHttpClient();
>>>> HttpContext localContext = new BasicHttpContext();
>>>>
>>>>
>>>> List<NameValuePair> qparams = new
>>>> ArrayList<NameValuePair>();
>>>> qparams.add(new BasicNameValuePair("txtUserName",
>>>> "myusername"));
>>>> qparams.add(new BasicNameValuePair("txtPassword",
>>>> "mypassword"));
>>>>
>>>> URI uri;
>>>> try {
>>>> uri = URIUtils.createURI("http", "my.website.com",
>>>> 8080, "/div/login",
>>>> URLEncodedUtils.format(qparams, "UTF-8"),
>>>> null);
>>>>
>>>> HttpPost httppost = new HttpPost(uri);
>>>> HttpResponse response = httpclient.execute(httppost,
>>>> localContext);
>>>> HttpEntity entity = response.getEntity();
>>>>
>>>> List<Cookie> cookies =
>>>> httpclient.getCookieStore().getCookies();
>>>>
>>>> if (cookies.isEmpty()) {
>>>> System.out.println("None");
>>>> } else {
>>>> for (int i = 0; i < cookies.size(); i++) {
>>>> System.out.println(i +" - " +
>>>> cookies.get(i).toString());
>>>> }
>>>> }
>>>>
>>>> if (entity != null) {
>>>> long len = entity.getContentLength();
>>>> if (len != -1) {
>>>> pageSource = EntityUtils.toString(entity);
>>>>
>>>> } else {
>>>> pageSource = "";
>>>> }
>>>> }
>>>>
>>>> } catch (URISyntaxException e) {
>>>> e.printStackTrace();
>>>> } catch (ClientProtocolException e) {
>>>> // TODO Auto-generated catch block
>>>> e.printStackTrace();
>>>> } catch (IOException e) {
>>>> // TODO Auto-generated catch block
>>>> e.printStackTrace();
>>>> }
>>>> return pageSource;
>>>> }
>>>> }
>>>>
>>>> The Form
>>>>
>>>>
>>>> ------------------------------------------------------------------------------------------------------
>>>>
>>>> <div class="loginPanel">
>>>> <div class="container">
>>>> <div class="title">Login</div>
>>>> <form id="loginForm" class="loginForm" method="POST"
>>>> action="/div/login">
>>>> <font class="portlet-msg-error"
>>>> style="font-weight: bold; font-size: 10px; color:#FF0000;
>>>> text-align: center;"></font>
>>>> <table>
>>>> <tr>
>>>> <td class="label">USERNAME:</td>
>>>> <td><input
>>>> value=""
>>>> class="edit"
>>>> id="txtUsernameLogin"
>>>> name="txtUserName"
>>>> type="text"/></td>
>>>> </tr>
>>>> <tr>
>>>> <td class="label">PASSWORD:</td>
>>>> <td><input
>>>> value=""
>>>> class="edit"
>>>> name="txtPassword"
>>>> type="password"/></td>
>>>> </tr>
>>>> <tr>
>>>> <td colspan="2" align="center">
>>>> <input type="checkbox" name="txtRemember"
>>>> value="true" align="middle"/>
>>>> Remember me on this computer
>>>> </td>
>>>> </tr>
>>>> <tr>
>>>> <td></td>
>>>> <td class=""><input type="submit"
>>>> class="submit"
>>>> value="Login"
>>>> alt="Login"/></td>
>>>> </tr>
>>>> </table>
>>>> </form>
>>>> </div>
>>>> </div>
>>>>
>>>> Any help will be greatly appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> Robert
>>>>
>>>>
>>> --------------------------------------------
>>> Ken Krugler
>>> +1 530-210-6378
>>> http://bixolabs.com
>>> e l a s t i c w e b m i n i n g
>>>
>>>
>>>
>>>
>>>
>>>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c w e b m i n i n g
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: Logging into a website issues with 4.0.1
Posted by Ken Krugler <kk...@transpac.com>.
On Feb 15, 2010, at 10:07am, Robert Stone wrote:
> Ken,
>
> They are in there. The problem is that when I send them from the
> script it
> is putting them in a different place then the manual. If you look at
> the
> wireshark log again you will see they are actually at the top for some
> reason not at the bottom like in the manual.
Those are URL parameters, not POST parameters.
See http://svn.apache.org/repos/asf/httpcomponents/httpclient/branches/4.0.x/httpclient/src/examples/org/apache/http/examples/client/ClientFormLogin.java
and specifically the httppost.setEntity() call.
-- Ken
> ------------------------------------------------------------------------------------------------------------------------------------
> The Automated Login
> -----------------------------------------------------------------------------------------------------------------------------------
>
>
> POST /div/login.jsp?txtUserName=myUserName&txtPassword=myPassWord
> HTTP/1.1
> <===== Here is where they are in the automated
>
> I don't know why this is because I think I am sending them properly.
> Here is
> the section from my code.
> List<NameValuePair> qparams = new
> ArrayList<NameValuePair>();
> qparams.add(new BasicNameValuePair("txtUserName",
> "myusername")); <== Here are the user name and password
> qparams.add(new BasicNameValuePair("txtPassword",
> "mypassword"));
>
> URI uri;
> try {
> uri = URIUtils.createURI("http",
> "my.website.com",
> 8080, "/div/login",
> URLEncodedUtils.format(qparams, "UTF-8"),
> null); <== Here is where I ad the params to the post
>
> HttpPost httppost = new HttpPost(uri);
> <===== and
> here is where I am sending the post.
>
> Any Ideas what is going wrong?
>
> Thanks,
>
> Robert
>
>
> On Mon, Feb 15, 2010 at 10:36 AM, Ken Krugler
> <kk...@transpac.com>wrote:
>
>> In the automated login, I don't see the POST parameters:
>>
>> txtUserName=myUserName&txtPassword=myPassWord
>>
>> -- Ken
>>
>> On Feb 15, 2010, at 7:36am, Robert Stone wrote:
>>
>> *Thanks to Jeff* for getting me on my way to figuring this issue
>> out. I
>>> took
>>>
>>> two wireshark logs to see what the differences are between the
>>> automated
>>> login and the manual login just as he suggested. The text of the
>>> wireshark
>>> logs are as follows.
>>>
>>> The Manual Login:
>>>
>>> -----------------------------------------------------------------------------------------------------------------------------
>>>
>>> POST /div/login HTTP/1.1
>>> Host: my.website.com:8080
>>> Connection: keep-alive
>>> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)
>>> AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5
>>> Referer: http://my.website.com:8080/div/login
>>> Content-Length: 38
>>> Cache-Control: max-age=0
>>> Origin: http://my.website.com:8080
>>> Content-Type: application/x-www-form-urlencoded
>>> Accept:
>>>
>>> application/xml,application/xhtml+xml,text/html;q=0.9,text/
>>> plain;q=0.8,image/png,*/*;q=0.5
>>> Accept-Encoding: gzip,deflate,sdch
>>> Cookie: JSESSIONID=AF810830BBC3D8869D70178E73239217
>>> Accept-Language: en-US,en;q=0.8
>>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
>>>
>>> txtUserName=myUserName&txtPassword=myPassWordHTTP/1.1 302 Moved
>>> Temporarily
>>> Server: Apache-Coyote/1.1
>>> Location: http://my.website.com:8080/div/
>>> Content-Length: 0
>>> Date: Fri, 12 Feb 2010 01:52:30 GMT
>>>
>>>
>>> ------------------------------------------------------------------------------------------------------------------------------------
>>> The Automated Login
>>>
>>> -----------------------------------------------------------------------------------------------------------------------------------
>>>
>>> POST /div/login.jsp?txtUserName=myUserName&txtPassword=myPassWord
>>> HTTP/1.1
>>> Content-Length: 0
>>> Host: my.website.com:8080
>>> Connection: Keep-Alive
>>> User-Agent: Apache-HttpClient/4.0.1 (java 1.5)
>>>
>>> HTTP/1.1 200 OK
>>> Server: Apache-Coyote/1.1
>>> Set-Cookie: JSESSIONID=9563FDBAA69961D49B607C63383C5F7A; Path=/div
>>> Cache-Control: no-cache, must-revalidate
>>> Expires: 0
>>> Date: Fri, 12 Feb 2010 17:09:24 GMT
>>> Content-Type: text/xml;charset=UTF-8
>>> Content-Length: 3648
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------
>>>
>>> As one can clearly see the Manual Login contains far much more
>>> information
>>> in the Packet. I have no idea how to include this information in the
>>> request
>>> that I am sending from the http-client class I made. I looked for
>>> methods
>>> with auto complete that would allow me to add this information
>>> however I
>>> found nothing specific. Can anyone get me started on the road to
>>> adding
>>> this
>>> information to my automated login's packet.
>>>
>>> Originally I attached my source code in a zip file to make the
>>> email more
>>> readable however this mail list doesn't seem to allow attachments
>>> so here
>>> is
>>> the source code in text format.
>>>
>>> Sorry in advance if it is ugly or hard to read.
>>>
>>> First the Java class
>>>
>>> -----------------------------------------------------------------------------------'
>>>
>>> import java.io.IOException;
>>> import java.net.URI;
>>> import java.net.URISyntaxException;
>>> import java.util.ArrayList;
>>> import java.util.List;
>>> import org.apache.http.HttpEntity;
>>> import org.apache.http.HttpResponse;
>>> import org.apache.http.NameValuePair;
>>> import org.apache.http.client.ClientProtocolException;
>>> import org.apache.http.client.entity.UrlEncodedFormEntity;
>>> import org.apache.http.client.methods.HttpGet;
>>> import org.apache.http.client.methods.HttpPost;
>>> import org.apache.http.client.utils.URIUtils;
>>> import org.apache.http.client.utils.URLEncodedUtils;
>>> import org.apache.http.cookie.Cookie;
>>> import org.apache.http.impl.client.DefaultHttpClient;
>>> import org.apache.http.message.BasicNameValuePair;
>>> import org.apache.http.protocol.BasicHttpContext;
>>> import org.apache.http.protocol.HTTP;
>>> import org.apache.http.protocol.HttpContext;
>>> import org.apache.http.util.EntityUtils;
>>>
>>> /**
>>> * WebUtils This class contains basic Web Utilities used for
>>> logging into
>>> and manipulating data
>>> * on a requested website.
>>> *
>>> * @author ME
>>> */
>>> public class WebUtils {
>>>
>>>
>>> /**
>>> * getPageSource This Method will return the source code of the
>>> requested page when given the
>>> * following parameters
>>> *
>>> * @return pageSource The source code of the requested page
>>> */
>>> public String getPageSource(){
>>>
>>> String pageSource = "";
>>>
>>> DefaultHttpClient httpclient = new
>>> DefaultHttpClient();
>>> HttpContext localContext = new BasicHttpContext();
>>>
>>>
>>> List<NameValuePair> qparams = new
>>> ArrayList<NameValuePair>();
>>> qparams.add(new BasicNameValuePair("txtUserName",
>>> "myusername"));
>>> qparams.add(new BasicNameValuePair("txtPassword",
>>> "mypassword"));
>>>
>>> URI uri;
>>> try {
>>> uri = URIUtils.createURI("http",
>>> "my.website.com",
>>> 8080, "/div/login",
>>> URLEncodedUtils.format(qparams, "UTF-8"),
>>> null);
>>>
>>> HttpPost httppost = new HttpPost(uri);
>>> HttpResponse response =
>>> httpclient.execute(httppost,
>>> localContext);
>>> HttpEntity entity = response.getEntity();
>>>
>>> List<Cookie> cookies =
>>> httpclient.getCookieStore().getCookies();
>>>
>>> if (cookies.isEmpty()) {
>>> System.out.println("None");
>>> } else {
>>> for (int i = 0; i < cookies.size(); i++) {
>>> System.out.println(i +" - " +
>>> cookies.get(i).toString());
>>> }
>>> }
>>>
>>> if (entity != null) {
>>> long len = entity.getContentLength();
>>> if (len != -1) {
>>> pageSource = EntityUtils.toString(entity);
>>>
>>> } else {
>>> pageSource = "";
>>> }
>>> }
>>>
>>> } catch (URISyntaxException e) {
>>> e.printStackTrace();
>>> } catch (ClientProtocolException e) {
>>> // TODO Auto-generated catch block
>>> e.printStackTrace();
>>> } catch (IOException e) {
>>> // TODO Auto-generated catch block
>>> e.printStackTrace();
>>> }
>>> return pageSource;
>>> }
>>> }
>>>
>>> The Form
>>>
>>> ------------------------------------------------------------------------------------------------------
>>>
>>> <div class="loginPanel">
>>> <div class="container">
>>> <div class="title">Login</div>
>>> <form id="loginForm" class="loginForm" method="POST"
>>> action="/div/login">
>>> <font class="portlet-msg-error"
>>> style="font-weight: bold; font-size: 10px;
>>> color:#FF0000;
>>> text-align: center;"></font>
>>> <table>
>>> <tr>
>>> <td class="label">USERNAME:</td>
>>> <td><input
>>> value=""
>>> class="edit"
>>> id="txtUsernameLogin"
>>> name="txtUserName"
>>> type="text"/></td>
>>> </tr>
>>> <tr>
>>> <td class="label">PASSWORD:</td>
>>> <td><input
>>> value=""
>>> class="edit"
>>> name="txtPassword"
>>> type="password"/></td>
>>> </tr>
>>> <tr>
>>> <td colspan="2" align="center">
>>> <input type="checkbox" name="txtRemember"
>>> value="true" align="middle"/>
>>> Remember me on this computer
>>> </td>
>>> </tr>
>>> <tr>
>>> <td></td>
>>> <td class=""><input type="submit"
>>> class="submit"
>>> value="Login"
>>> alt="Login"/></td>
>>> </tr>
>>> </table>
>>> </form>
>>> </div>
>>> </div>
>>>
>>> Any help will be greatly appreciated.
>>>
>>> Thanks,
>>>
>>> Robert
>>>
>>
>> --------------------------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://bixolabs.com
>> e l a s t i c w e b m i n i n g
>>
>>
>>
>>
>>
--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Logging into a website issues with 4.0.1
Posted by Robert Stone <rt...@gmail.com>.
Ken,
They are in there. The problem is that when I send them from the script it
is putting them in a different place then the manual. If you look at the
wireshark log again you will see they are actually at the top for some
reason not at the bottom like in the manual.
------------------------------------------------------------------------------------------------------------------------------------
The Automated Login
-----------------------------------------------------------------------------------------------------------------------------------
POST /div/login.jsp?txtUserName=myUserName&txtPassword=myPassWord HTTP/1.1
<===== Here is where they are in the automated
I don't know why this is because I think I am sending them properly. Here is
the section from my code.
List<NameValuePair> qparams = new
ArrayList<NameValuePair>();
qparams.add(new BasicNameValuePair("txtUserName",
"myusername")); <== Here are the user name and password
qparams.add(new BasicNameValuePair("txtPassword",
"mypassword"));
URI uri;
try {
uri = URIUtils.createURI("http", "my.website.com",
8080, "/div/login",
URLEncodedUtils.format(qparams, "UTF-8"),
null); <== Here is where I ad the params to the post
HttpPost httppost = new HttpPost(uri); <===== and
here is where I am sending the post.
Any Ideas what is going wrong?
Thanks,
Robert
On Mon, Feb 15, 2010 at 10:36 AM, Ken Krugler
<kk...@transpac.com>wrote:
> In the automated login, I don't see the POST parameters:
>
> txtUserName=myUserName&txtPassword=myPassWord
>
> -- Ken
>
> On Feb 15, 2010, at 7:36am, Robert Stone wrote:
>
> *Thanks to Jeff* for getting me on my way to figuring this issue out. I
>> took
>>
>> two wireshark logs to see what the differences are between the automated
>> login and the manual login just as he suggested. The text of the wireshark
>> logs are as follows.
>>
>> The Manual Login:
>>
>> -----------------------------------------------------------------------------------------------------------------------------
>>
>> POST /div/login HTTP/1.1
>> Host: my.website.com:8080
>> Connection: keep-alive
>> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)
>> AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5
>> Referer: http://my.website.com:8080/div/login
>> Content-Length: 38
>> Cache-Control: max-age=0
>> Origin: http://my.website.com:8080
>> Content-Type: application/x-www-form-urlencoded
>> Accept:
>>
>> application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
>> Accept-Encoding: gzip,deflate,sdch
>> Cookie: JSESSIONID=AF810830BBC3D8869D70178E73239217
>> Accept-Language: en-US,en;q=0.8
>> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
>>
>> txtUserName=myUserName&txtPassword=myPassWordHTTP/1.1 302 Moved
>> Temporarily
>> Server: Apache-Coyote/1.1
>> Location: http://my.website.com:8080/div/
>> Content-Length: 0
>> Date: Fri, 12 Feb 2010 01:52:30 GMT
>>
>>
>> ------------------------------------------------------------------------------------------------------------------------------------
>> The Automated Login
>>
>> -----------------------------------------------------------------------------------------------------------------------------------
>>
>> POST /div/login.jsp?txtUserName=myUserName&txtPassword=myPassWord HTTP/1.1
>> Content-Length: 0
>> Host: my.website.com:8080
>> Connection: Keep-Alive
>> User-Agent: Apache-HttpClient/4.0.1 (java 1.5)
>>
>> HTTP/1.1 200 OK
>> Server: Apache-Coyote/1.1
>> Set-Cookie: JSESSIONID=9563FDBAA69961D49B607C63383C5F7A; Path=/div
>> Cache-Control: no-cache, must-revalidate
>> Expires: 0
>> Date: Fri, 12 Feb 2010 17:09:24 GMT
>> Content-Type: text/xml;charset=UTF-8
>> Content-Length: 3648
>>
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> As one can clearly see the Manual Login contains far much more information
>> in the Packet. I have no idea how to include this information in the
>> request
>> that I am sending from the http-client class I made. I looked for methods
>> with auto complete that would allow me to add this information however I
>> found nothing specific. Can anyone get me started on the road to adding
>> this
>> information to my automated login's packet.
>>
>> Originally I attached my source code in a zip file to make the email more
>> readable however this mail list doesn't seem to allow attachments so here
>> is
>> the source code in text format.
>>
>> Sorry in advance if it is ugly or hard to read.
>>
>> First the Java class
>>
>> -----------------------------------------------------------------------------------'
>>
>> import java.io.IOException;
>> import java.net.URI;
>> import java.net.URISyntaxException;
>> import java.util.ArrayList;
>> import java.util.List;
>> import org.apache.http.HttpEntity;
>> import org.apache.http.HttpResponse;
>> import org.apache.http.NameValuePair;
>> import org.apache.http.client.ClientProtocolException;
>> import org.apache.http.client.entity.UrlEncodedFormEntity;
>> import org.apache.http.client.methods.HttpGet;
>> import org.apache.http.client.methods.HttpPost;
>> import org.apache.http.client.utils.URIUtils;
>> import org.apache.http.client.utils.URLEncodedUtils;
>> import org.apache.http.cookie.Cookie;
>> import org.apache.http.impl.client.DefaultHttpClient;
>> import org.apache.http.message.BasicNameValuePair;
>> import org.apache.http.protocol.BasicHttpContext;
>> import org.apache.http.protocol.HTTP;
>> import org.apache.http.protocol.HttpContext;
>> import org.apache.http.util.EntityUtils;
>>
>> /**
>> * WebUtils This class contains basic Web Utilities used for logging into
>> and manipulating data
>> * on a requested website.
>> *
>> * @author ME
>> */
>> public class WebUtils {
>>
>>
>> /**
>> * getPageSource This Method will return the source code of the
>> requested page when given the
>> * following parameters
>> *
>> * @return pageSource The source code of the requested page
>> */
>> public String getPageSource(){
>>
>> String pageSource = "";
>>
>> DefaultHttpClient httpclient = new DefaultHttpClient();
>> HttpContext localContext = new BasicHttpContext();
>>
>>
>> List<NameValuePair> qparams = new
>> ArrayList<NameValuePair>();
>> qparams.add(new BasicNameValuePair("txtUserName",
>> "myusername"));
>> qparams.add(new BasicNameValuePair("txtPassword",
>> "mypassword"));
>>
>> URI uri;
>> try {
>> uri = URIUtils.createURI("http", "my.website.com",
>> 8080, "/div/login",
>> URLEncodedUtils.format(qparams, "UTF-8"),
>> null);
>>
>> HttpPost httppost = new HttpPost(uri);
>> HttpResponse response = httpclient.execute(httppost,
>> localContext);
>> HttpEntity entity = response.getEntity();
>>
>> List<Cookie> cookies =
>> httpclient.getCookieStore().getCookies();
>>
>> if (cookies.isEmpty()) {
>> System.out.println("None");
>> } else {
>> for (int i = 0; i < cookies.size(); i++) {
>> System.out.println(i +" - " +
>> cookies.get(i).toString());
>> }
>> }
>>
>> if (entity != null) {
>> long len = entity.getContentLength();
>> if (len != -1) {
>> pageSource = EntityUtils.toString(entity);
>>
>> } else {
>> pageSource = "";
>> }
>> }
>>
>> } catch (URISyntaxException e) {
>> e.printStackTrace();
>> } catch (ClientProtocolException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> } catch (IOException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>> return pageSource;
>> }
>> }
>>
>> The Form
>>
>> ------------------------------------------------------------------------------------------------------
>>
>> <div class="loginPanel">
>> <div class="container">
>> <div class="title">Login</div>
>> <form id="loginForm" class="loginForm" method="POST"
>> action="/div/login">
>> <font class="portlet-msg-error"
>> style="font-weight: bold; font-size: 10px; color:#FF0000;
>> text-align: center;"></font>
>> <table>
>> <tr>
>> <td class="label">USERNAME:</td>
>> <td><input
>> value=""
>> class="edit"
>> id="txtUsernameLogin"
>> name="txtUserName"
>> type="text"/></td>
>> </tr>
>> <tr>
>> <td class="label">PASSWORD:</td>
>> <td><input
>> value=""
>> class="edit"
>> name="txtPassword"
>> type="password"/></td>
>> </tr>
>> <tr>
>> <td colspan="2" align="center">
>> <input type="checkbox" name="txtRemember"
>> value="true" align="middle"/>
>> Remember me on this computer
>> </td>
>> </tr>
>> <tr>
>> <td></td>
>> <td class=""><input type="submit"
>> class="submit"
>> value="Login"
>> alt="Login"/></td>
>> </tr>
>> </table>
>> </form>
>> </div>
>> </div>
>>
>> Any help will be greatly appreciated.
>>
>> Thanks,
>>
>> Robert
>>
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c w e b m i n i n g
>
>
>
>
>
Re: Logging into a website issues with 4.0.1
Posted by Ken Krugler <kk...@transpac.com>.
In the automated login, I don't see the POST parameters:
txtUserName=myUserName&txtPassword=myPassWord
-- Ken
On Feb 15, 2010, at 7:36am, Robert Stone wrote:
> *Thanks to Jeff* for getting me on my way to figuring this issue
> out. I took
> two wireshark logs to see what the differences are between the
> automated
> login and the manual login just as he suggested. The text of the
> wireshark
> logs are as follows.
>
> The Manual Login:
> -----------------------------------------------------------------------------------------------------------------------------
>
> POST /div/login HTTP/1.1
> Host: my.website.com:8080
> Connection: keep-alive
> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)
> AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5
> Referer: http://my.website.com:8080/div/login
> Content-Length: 38
> Cache-Control: max-age=0
> Origin: http://my.website.com:8080
> Content-Type: application/x-www-form-urlencoded
> Accept:
> application/xml,application/xhtml+xml,text/html;q=0.9,text/
> plain;q=0.8,image/png,*/*;q=0.5
> Accept-Encoding: gzip,deflate,sdch
> Cookie: JSESSIONID=AF810830BBC3D8869D70178E73239217
> Accept-Language: en-US,en;q=0.8
> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
>
> txtUserName=myUserName&txtPassword=myPassWordHTTP/1.1 302 Moved
> Temporarily
> Server: Apache-Coyote/1.1
> Location: http://my.website.com:8080/div/
> Content-Length: 0
> Date: Fri, 12 Feb 2010 01:52:30 GMT
>
> ------------------------------------------------------------------------------------------------------------------------------------
> The Automated Login
> -----------------------------------------------------------------------------------------------------------------------------------
>
> POST /div/login.jsp?txtUserName=myUserName&txtPassword=myPassWord
> HTTP/1.1
> Content-Length: 0
> Host: my.website.com:8080
> Connection: Keep-Alive
> User-Agent: Apache-HttpClient/4.0.1 (java 1.5)
>
> HTTP/1.1 200 OK
> Server: Apache-Coyote/1.1
> Set-Cookie: JSESSIONID=9563FDBAA69961D49B607C63383C5F7A; Path=/div
> Cache-Control: no-cache, must-revalidate
> Expires: 0
> Date: Fri, 12 Feb 2010 17:09:24 GMT
> Content-Type: text/xml;charset=UTF-8
> Content-Length: 3648
>
> ----------------------------------------------------------------------------------------------------
>
> As one can clearly see the Manual Login contains far much more
> information
> in the Packet. I have no idea how to include this information in the
> request
> that I am sending from the http-client class I made. I looked for
> methods
> with auto complete that would allow me to add this information
> however I
> found nothing specific. Can anyone get me started on the road to
> adding this
> information to my automated login's packet.
>
> Originally I attached my source code in a zip file to make the email
> more
> readable however this mail list doesn't seem to allow attachments so
> here is
> the source code in text format.
>
> Sorry in advance if it is ugly or hard to read.
>
> First the Java class
> -----------------------------------------------------------------------------------'
>
> import java.io.IOException;
> import java.net.URI;
> import java.net.URISyntaxException;
> import java.util.ArrayList;
> import java.util.List;
> import org.apache.http.HttpEntity;
> import org.apache.http.HttpResponse;
> import org.apache.http.NameValuePair;
> import org.apache.http.client.ClientProtocolException;
> import org.apache.http.client.entity.UrlEncodedFormEntity;
> import org.apache.http.client.methods.HttpGet;
> import org.apache.http.client.methods.HttpPost;
> import org.apache.http.client.utils.URIUtils;
> import org.apache.http.client.utils.URLEncodedUtils;
> import org.apache.http.cookie.Cookie;
> import org.apache.http.impl.client.DefaultHttpClient;
> import org.apache.http.message.BasicNameValuePair;
> import org.apache.http.protocol.BasicHttpContext;
> import org.apache.http.protocol.HTTP;
> import org.apache.http.protocol.HttpContext;
> import org.apache.http.util.EntityUtils;
>
> /**
> * WebUtils This class contains basic Web Utilities used for logging
> into
> and manipulating data
> * on a requested website.
> *
> * @author ME
> */
> public class WebUtils {
>
>
> /**
> * getPageSource This Method will return the source code of the
> requested page when given the
> * following parameters
> *
> * @return pageSource The source code of the requested page
> */
> public String getPageSource(){
>
> String pageSource = "";
>
> DefaultHttpClient httpclient = new
> DefaultHttpClient();
> HttpContext localContext = new BasicHttpContext();
>
>
> List<NameValuePair> qparams = new
> ArrayList<NameValuePair>();
> qparams.add(new BasicNameValuePair("txtUserName",
> "myusername"));
> qparams.add(new BasicNameValuePair("txtPassword",
> "mypassword"));
>
> URI uri;
> try {
> uri = URIUtils.createURI("http",
> "my.website.com",
> 8080, "/div/login",
> URLEncodedUtils.format(qparams, "UTF-8"),
> null);
>
> HttpPost httppost = new HttpPost(uri);
> HttpResponse response =
> httpclient.execute(httppost,
> localContext);
> HttpEntity entity = response.getEntity();
>
> List<Cookie> cookies =
> httpclient.getCookieStore().getCookies();
>
> if (cookies.isEmpty()) {
> System.out.println("None");
> } else {
> for (int i = 0; i < cookies.size(); i++) {
> System.out.println(i +" - " +
> cookies.get(i).toString());
> }
> }
>
> if (entity != null) {
> long len = entity.getContentLength();
> if (len != -1) {
> pageSource = EntityUtils.toString(entity);
>
> } else {
> pageSource = "";
> }
> }
>
> } catch (URISyntaxException e) {
> e.printStackTrace();
> } catch (ClientProtocolException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> } catch (IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> return pageSource;
> }
> }
>
> The Form
> ------------------------------------------------------------------------------------------------------
>
> <div class="loginPanel">
> <div class="container">
> <div class="title">Login</div>
> <form id="loginForm" class="loginForm" method="POST"
> action="/div/login">
> <font class="portlet-msg-error"
> style="font-weight: bold; font-size: 10px;
> color:#FF0000;
> text-align: center;"></font>
> <table>
> <tr>
> <td class="label">USERNAME:</td>
> <td><input
> value=""
> class="edit"
> id="txtUsernameLogin"
> name="txtUserName"
> type="text"/></td>
> </tr>
> <tr>
> <td class="label">PASSWORD:</td>
> <td><input
> value=""
> class="edit"
> name="txtPassword"
> type="password"/></td>
> </tr>
> <tr>
> <td colspan="2" align="center">
> <input type="checkbox" name="txtRemember"
> value="true" align="middle"/>
> Remember me on this computer
> </td>
> </tr>
> <tr>
> <td></td>
> <td class=""><input type="submit"
> class="submit"
> value="Login"
> alt="Login"/></td>
> </tr>
> </table>
> </form>
> </div>
> </div>
>
> Any help will be greatly appreciated.
>
> Thanks,
>
> Robert
--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g