You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Sanjeev Kumar Neemkar <sa...@gmail.com> on 2006/02/18 01:30:04 UTC
HTTPClient Redirection Problem
Hi ,
Requirement: We want to simulate following scenario with HttpClient.
Log on to website which requires Digital Certificate.
After that click on a link of Welcome Page and get the response and based on
new page again do some
processing on the same session.
In another sense we can say Text based browser.
Current Status of Solution:
I have written a small piece of code for two-factor authentication using
apache
HTTPClient and AuthSSLProtocolSocketFactory as per SSL Guide.
Authentication is successful by passing client certificate with private key
and server's truststore and respective password. Response from the server is
chunk of html (Welcome/Home Page) .
My requirement is, to follow the hyper-link present in the response html
(Welcome/Home Page). I mean i want to click on test-hyper-link and get the
response.
Is there any way to do this ?
Please find the java code used below.
base url : https://abcdev.test.com
Link URL on Welcome Page:
https://abcdev.test.com/ABCProject/servlet/PageDirector?choice=upload&userRole=ABCUser
It can be Get Method or Post Method in real scenario.
We tried with relative, compelete URL also. Also we passed Name Value pair.
Our current understanding is we have to use same method which we used for
first request/login.
Becuase it is storing the actual connection and session.
-Sanjeev
JAVA CODE USED:
-------------------------
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URL;
import java.util.Properties;
import org.apache.commons.httpclient.Credentials;
import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;
import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpClient ;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.HttpRecoverableException;
import org.apache.commons.httpclient.HttpVersion ;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.params.HostParams ;
import org.apache.commons.httpclient.params.HttpConnectionParams;
import org.apache.commons.httpclient.params.HttpMethodParams;
import org.apache.commons.httpclient.protocol.Protocol;
import org.apache.log4j.Logger ;
import org.apache.log4j.PropertyConfigurator;
/**
* Client Application is command line version of the Web based.
*
*/
public class ABCCLi {
/**
* static variable to hold the logger class name.
*/
private static Logger ourLogger = Logger
.getLogger(com.adp.ABC.cli.ABCCLi.class);
private static String strClientKeyStore;
private static String strClientKeyStorePassword;
private static String strServerTrustStore;
private static String strServerTrustStorePassword;
private static String strABCServerURL;
private static String strABCUserName;
private static String strABCPassword;
public static final int HTTP_FAILURE = -1;
public static final int HTTP_SUCCESS = 200;
public static final int HTTP_AUTHENTICATION_ERROR = 401;
public static final int HTTP_FILE_NOT_FOUND = 404;
private static final int RETRY_COUNT = 5;
private int statusCode = HTTP_FAILURE;
private static Protocol authhttps, easyhttps;
private String lastModifiedDate = null;
/**
* main entry for the application
*
* @param args
* @throws Exception
*/
public static void main(String[] args) throws Exception {
//load the log4j config
PropertyConfigurator.configure(args[1]);
ourLogger.info("java.home:" + System.getProperty("java.home"));
ourLogger.info("java.library.path:"
+ System.getProperty("java.library.path"));
ourLogger.info ("java.endorsed.dirs:"
+ System.getProperty("java.endorsed.dirs"));
ourLogger.info("sun.boot.library.path:"
+ System.getProperty("sun.boot.library.path"));
ourLogger.info ("java.ext.dirs:" + System.getProperty("java.ext.dirs"));
//load and set the properties
setProperties(args[0]);
ABCCLi ABCcli = new ABCCLi();
String msg = new String(ABCcli.getResponse(strABCServerURL,
strABCUserName, strABCPassword));
//ourLogger.info("Message::>" + msg);
}
/**
* Default Constructor. Registering of the protocol occures here.
*
*/
public ABCCLi() {
try {
authhttps = new Protocol("https",
new AuthSSLProtocolSocketFactory(new URL("file:"
+ strClientKeyStore), strClientKeyStorePassword,
new URL("file:" + strServerTrustStore),
strServerTrustStorePassword), 443);
Protocol.registerProtocol("https", authhttps);
} catch (Exception ex) {
ourLogger.error("Error while registering Protocol", ex);
ex.printStackTrace();
}
}
/**
* This method returns the response in the form of byte array.
*
* @param url -
* Secure URL of the server to be connected.
* @return byte array of the response
* @throws Exception
*/
public byte[] getResponse(String url) throws Exception {
return getResponse(url, null, null);
}
/**
* This method returns the response in the form of byte array given
username
* and password.
*
* @param url -
* Secure URL of the server to be connected.
* @param username -
* User name for the secure site
* @param password -
* Password for the secure site
* @return byte array of the response
* @throws Exception
*/
public byte[] getResponse(String url, String username, String password)
throws Exception {
String SMSESSION = null;
String ABCJSESSIONID = null;
String SMCHALLENGE = null;
String SMIDENTITY = null;
// Create an instance of HttpClient.
HttpClient client = new HttpClient();
HttpVersion ver = (HttpVersion) client.getParams().getParameter(
"http.protocol.version");
ourLogger.info("----------------- Http Version : " + ver.getMajor()
+ "." + ver.getMinor());
client.getHttpConnectionManager().getParams().setBooleanParameter(
"http.connection.stalecheck", true);
if (username != null) {
// pass our credentials to HttpClient, they will only be used for
// authenticating to servers with realm "realm", to authenticate
// against an arbitrary realm change this to null.
client.getParams().setAuthenticationPreemptive(true);
Credentials defaultcreds = new UsernamePasswordCredentials(
username, password);
client.getState().setCredentials(
new AuthScope(AuthScope.ANY_HOST , AuthScope.ANY_PORT,
AuthScope.ANY_REALM), defaultcreds);
}
// Execute the method.
byte[] responseBody = null;
//Perform Recoverable Retry
for (int i = 0; statusCode == -1 && i < RETRY_COUNT; ++i) {
// Create a method instance.
// url = https://abcdev.test.com
HttpMethod method = new GetMethod(url);
client.getHostConfiguration().setParams(new HostParams());
method.setRequestHeader("Connection", "Keep-Alive");
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(5, false));
method.setDoAuthentication(true);
method.setFollowRedirects(true);
try {
HttpConnectionParams httpConnectionParams = new HttpConnectionParams();
httpConnectionParams.setSoTimeout(10 * 1000);
ourLogger.info("executing method client.executeMethod(method)");
// execute the method.
statusCode = client.executeMethod(method);
ourLogger.debug("statusCode = " + statusCode);
responseBody = method.getResponseBody();
Header lastModifiedHdr = method
.getResponseHeader("Last-Modified");
if (lastModifiedHdr != null)
lastModifiedDate = (String) lastModifiedHdr.getValue();
ourLogger.info("Clicking on link of welcome page....");
// We tried with relative and compelte URL also,
// Tested with - add as Name Value Pair for parameters also
// It is not working
method.setPath
("/ABCProject/servlet/PageDirector?choice=upload&userRole=ABCUser");
method.setRequestHeader("Connection", "Keep-Alive");
statusCode = client.executeMethod (method);
ourLogger.debug("statusCode = " + statusCode);
responseBody = method.getResponseBody();
return responseBody;
} catch (HttpRecoverableException e) {
if (2 >= 1) {
e.printStackTrace();
throw new Exception("ABCCLi.getResponse.unRecovered ");
} else {
ourLogger
.error("A recoverable exception occurred, retrying."
+ e.getMessage());
}
} catch (HttpException e) {
ourLogger.error("Failed to access the url " + url + ": "
+ e.getMessage());
e.printStackTrace();
throw new Exception("ABCCLi.getResponse.HttpException");
} catch (IOException e) {
ourLogger.error("Failed to access the url " + url + ": "
+ e.getMessage());
e.printStackTrace();
throw new Exception("ABCCLi.getResponse.IOException");
} finally {
// Release the connection.
method.releaseConnection();
}
}
return responseBody;
}
/**
* Returns the status code
*
* @return status code
*/
public int getStatusCode() {
return statusCode;
}
/**
* Returns the last modified date
*
* @return last modified date
*/
public String getLastModifiedDate() {
return lastModifiedDate;
}
/**
* Method to perform the loading of the properties from the properties file
* and sets to the class level variables.
*
* @param pPropertyFile
*/
private static void setProperties(String pPropertyFile) {
ourLogger.info("Loading Properties file & setting to member variables");
Properties prop;
FileInputStream fis;
try {
fis = new FileInputStream(pPropertyFile);
prop = new Properties();
prop.load(fis);
//Fetching values from the properties file
strClientKeyStore = prop.getProperty("client_keystore");
ourLogger.debug("strClientKeyStore=" + strClientKeyStore);
strClientKeyStorePassword = prop
.getProperty("client_keystore_password");
ourLogger.debug("strClientKeyStorePassword="
+ strClientKeyStorePassword);
strServerTrustStore = prop.getProperty("server_certificate");
ourLogger.debug("strServerTrustStore=" + strServerTrustStore);
strServerTrustStorePassword = prop
.getProperty("server_truststore_password");
ourLogger.debug("strServerTrustStorePassword="
+ strServerTrustStorePassword);
// URL is https://abcdev.test.com
strABCServerURL = prop.getProperty("ABC_server_url");
ourLogger.debug("strABCServerURL=" + strABCServerURL);
strABCUserName = prop.getProperty("user_name");
ourLogger.debug("strABCUserName=" + strABCUserName);
strABCPassword = prop.getProperty("password");
ourLogger.debug("strABCPassword=" + strABCPassword);
} catch (FileNotFoundException fileNotFoundExp) {
ourLogger.error("FileNotFoundException() : " + fileNotFoundExp);
} catch (IOException ioExp) {
ourLogger.error("IOException : " + ioExp);
} catch (NullPointerException nullPointerExp) {
ourLogger.error("NullPointerException : " + nullPointerExp);
} catch (Exception excep) {
ourLogger.error("Exception : " + excep);
}
}
}
Re: HTTPClient Redirection Problem
Posted by sebb <se...@gmail.com>.
On 22/02/06, Sanjeev Kumar Neemkar <sa...@gmail.com> wrote:
> Hello Roland,
>
> Considering URL-1 as http://www.apache.org/ and URL-2 as
> http://www.apache.org/foundation/faq.html, below is the piece of code.
> As per the steps and advice, which i followed as said, but it is again
> giving ResponseBody for URL-2 as the that of URL-1.
> Am i missing something, please correct me.
>
> Requirement: URL-2 is embeded in the page of URL-1. I need to simulate
> the hyper-link(URL-2) clicking in the page of URL-1 using HTTPClient.
>
> Please help,
>
> Thanks in Advance,
> Sanjeev Kumar Neemkar
>
> Java Code used as below:
> ******************************
>
> import java.io.IOException;
>
> import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;
> import org.apache.commons.httpclient.HttpClient;
> import org.apache.commons.httpclient.HttpException;
> import org.apache.commons.httpclient.HttpStatus;
> import org.apache.commons.httpclient.methods.GetMethod;
> import org.apache.commons.httpclient.params.HttpMethodParams;
>
> public class HttpClientTutorial {
>
> private static String url = "http://www.apache.org/";
>
> public static void main(String[] args) {
> // Create an instance of HttpClient.
> HttpClient client = new HttpClient();
>
> // Create a method instance.
> GetMethod method1 = new GetMethod(url); //URL-1
> //http://www.apache.org/foundation/faq.html
> GetMethod method2 = new GetMethod(url + "foundation/faq.html"); //URL-2
>
> // Provide custom retry handler is necessary
> method1.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
> new DefaultHttpMethodRetryHandler(3, false));
> // Provide custom retry handler is necessary
> method2.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
> new DefaultHttpMethodRetryHandler(3, false));
>
> try {
> // Execute the method.
> int statusCode = client.executeMethod(method1);
>
> if (statusCode != HttpStatus.SC_OK) {
> System.err.println("Method1 failed: " + method1.getStatusLine());
> }
>
> // Read the response body.
> byte[] responseBody1 = method1.getResponseBody();
> //Release the connection.
> method1.releaseConnection();
> // Deal with the response.
> // Use caution: ensure correct character encoding and is not binary
> // data
> System.out.println("URL-1::" + new String(responseBody1));
>
> ////////////////////URL-2/////////////////////
> // Execute the method.
> int statusCode2 = client.executeMethod(method2);
>
> if (statusCode2 != HttpStatus.SC_OK) {
> System.err.println("Method2 failed: " + method2.getStatusLine());
> }
>
> // Read the response body.
> byte[] responseBody2 = method1.getResponseBody();
^^^^^^^^^^^
> //Release the connection.
> method2.releaseConnection();
> // Deal with the response.
> // Use caution: ensure correct character encoding and is not binary
> // data
> System.out.println("URL-2::" + new String(responseBody2));
>
> } catch (HttpException e) {
> System.err.println("Fatal protocol violation: " + e.getMessage());
> e.printStackTrace();
> } catch (IOException e) {
> System.err.println("Fatal transport error: " + e.getMessage());
> e.printStackTrace();
> } finally {
> // Release the connection.
> //method1.releaseConnection();
> }
> }
> }
>
> .......................................................................................................................................................................................................
> On 2/21/06, Roland Weber <ht...@dubioso.net> wrote:
> >
> > Hello Sanjeev,
> >
> > > My requirement is, to follow the hyper-link present in the response html
> > > (Welcome/Home Page). I mean i want to click on test-hyper-link and get
> > the
> > > response.
> > >
> > > Is there any way to do this ?
> >
> > You've got the page. Scan it for the hyperlink to get the new URL.
> >
> > > Link URL on Welcome Page:
> > >
> > https://abcdev.test.com/ABCProject/servlet/PageDirector?choice=upload&userRole=ABCUser
> >
> > That link looks so static you can probably hard-wire it into your
> > application.
> >
> > > It can be Get Method or Post Method in real scenario.
> >
> > How is that? It's either the action attribute of a form, then the FORM
> > says
> > which one it is. Or it is not in a form, then it is a GET request. If the
> > same URL is accessed both ways, then the server doesn't care which one it
> > is.
> >
> > > Our current understanding is we have to use same method which we used
> > for
> > > first request/login.
> >
> > If you mean "method object", then you are wrong. HttpClient method objects
> > can not be re-used, or at least should not. There were some methods for
> > re-using, but it is rather pointless to try. Just create a new one.
> >
> > > Becuase it is storing the actual connection and session.
> >
> > The connection is kept in the method so you can release it when you're
> > done with the response. The connection manager will try to keep the
> > connection alive for the next request, but that is strictly a performance
> > optimization.
> > The session is in no way related to a connection. Session cookies are
> > stored
> > in the HttpState. If you didn't create an extra HttpState, the default
> > state
> > in the HttpClient will be used.
> >
> > I recommend that you create a new method object to access the second link,
> > and execute it with the same HttpClient object as the first one. Don't
> > forget to release the connection after each request, but not before you're
> > done with reading the response.
> >
> > hope that helps,
> > Roland
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> >
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: HTTPClient Redirection Problem
Posted by Sanjeev Kumar Neemkar <sa...@gmail.com>.
Thank you , It's working now.
-Sanjeev
On 2/21/06, Roland Weber <RO...@de.ibm.com> wrote:
>
> Just in case you missed sebb's point due to proportional font issues...
>
> > As per the steps and advice, which i followed as said, but it is
> again
> > giving ResponseBody for URL-2 as the that of URL-1.
> > [...]
> >
> > byte[] responseBody2 = method1.getResponseBody();
>
> cheers,
> Roland
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
Re: HTTPClient Redirection Problem
Posted by Roland Weber <RO...@de.ibm.com>.
Just in case you missed sebb's point due to proportional font issues...
> As per the steps and advice, which i followed as said, but it is
again
> giving ResponseBody for URL-2 as the that of URL-1.
> [...]
>
> byte[] responseBody2 = method1.getResponseBody();
cheers,
Roland
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: HTTPClient Redirection Problem
Posted by Sanjeev Kumar Neemkar <sa...@gmail.com>.
Hello Roland,
Considering URL-1 as http://www.apache.org/ and URL-2 as
http://www.apache.org/foundation/faq.html, below is the piece of code.
As per the steps and advice, which i followed as said, but it is again
giving ResponseBody for URL-2 as the that of URL-1.
Am i missing something, please correct me.
Requirement: URL-2 is embeded in the page of URL-1. I need to simulate
the hyper-link(URL-2) clicking in the page of URL-1 using HTTPClient.
Please help,
Thanks in Advance,
Sanjeev Kumar Neemkar
Java Code used as below:
******************************
import java.io.IOException;
import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.params.HttpMethodParams;
public class HttpClientTutorial {
private static String url = "http://www.apache.org/";
public static void main(String[] args) {
// Create an instance of HttpClient.
HttpClient client = new HttpClient();
// Create a method instance.
GetMethod method1 = new GetMethod(url); //URL-1
//http://www.apache.org/foundation/faq.html
GetMethod method2 = new GetMethod(url + "foundation/faq.html"); //URL-2
// Provide custom retry handler is necessary
method1.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
// Provide custom retry handler is necessary
method2.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
try {
// Execute the method.
int statusCode = client.executeMethod(method1);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method1 failed: " + method1.getStatusLine());
}
// Read the response body.
byte[] responseBody1 = method1.getResponseBody();
//Release the connection.
method1.releaseConnection();
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary
// data
System.out.println("URL-1::" + new String(responseBody1));
////////////////////URL-2/////////////////////
// Execute the method.
int statusCode2 = client.executeMethod(method2);
if (statusCode2 != HttpStatus.SC_OK) {
System.err.println("Method2 failed: " + method2.getStatusLine());
}
// Read the response body.
byte[] responseBody2 = method1.getResponseBody();
//Release the connection.
method2.releaseConnection();
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary
// data
System.out.println("URL-2::" + new String(responseBody2));
} catch (HttpException e) {
System.err.println("Fatal protocol violation: " + e.getMessage());
e.printStackTrace();
} catch (IOException e) {
System.err.println("Fatal transport error: " + e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
//method1.releaseConnection();
}
}
}
.......................................................................................................................................................................................................
On 2/21/06, Roland Weber <ht...@dubioso.net> wrote:
>
> Hello Sanjeev,
>
> > My requirement is, to follow the hyper-link present in the response html
> > (Welcome/Home Page). I mean i want to click on test-hyper-link and get
> the
> > response.
> >
> > Is there any way to do this ?
>
> You've got the page. Scan it for the hyperlink to get the new URL.
>
> > Link URL on Welcome Page:
> >
> https://abcdev.test.com/ABCProject/servlet/PageDirector?choice=upload&userRole=ABCUser
>
> That link looks so static you can probably hard-wire it into your
> application.
>
> > It can be Get Method or Post Method in real scenario.
>
> How is that? It's either the action attribute of a form, then the FORM
> says
> which one it is. Or it is not in a form, then it is a GET request. If the
> same URL is accessed both ways, then the server doesn't care which one it
> is.
>
> > Our current understanding is we have to use same method which we used
> for
> > first request/login.
>
> If you mean "method object", then you are wrong. HttpClient method objects
> can not be re-used, or at least should not. There were some methods for
> re-using, but it is rather pointless to try. Just create a new one.
>
> > Becuase it is storing the actual connection and session.
>
> The connection is kept in the method so you can release it when you're
> done with the response. The connection manager will try to keep the
> connection alive for the next request, but that is strictly a performance
> optimization.
> The session is in no way related to a connection. Session cookies are
> stored
> in the HttpState. If you didn't create an extra HttpState, the default
> state
> in the HttpClient will be used.
>
> I recommend that you create a new method object to access the second link,
> and execute it with the same HttpClient object as the first one. Don't
> forget to release the connection after each request, but not before you're
> done with reading the response.
>
> hope that helps,
> Roland
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
Re: HTTPClient Redirection Problem
Posted by Roland Weber <ht...@dubioso.net>.
Hello Sanjeev,
> My requirement is, to follow the hyper-link present in the response html
> (Welcome/Home Page). I mean i want to click on test-hyper-link and get the
> response.
>
> Is there any way to do this ?
You've got the page. Scan it for the hyperlink to get the new URL.
> Link URL on Welcome Page:
> https://abcdev.test.com/ABCProject/servlet/PageDirector?choice=upload&userRole=ABCUser
That link looks so static you can probably hard-wire it into your application.
> It can be Get Method or Post Method in real scenario.
How is that? It's either the action attribute of a form, then the FORM says
which one it is. Or it is not in a form, then it is a GET request. If the
same URL is accessed both ways, then the server doesn't care which one it is.
> Our current understanding is we have to use same method which we used for
> first request/login.
If you mean "method object", then you are wrong. HttpClient method objects
can not be re-used, or at least should not. There were some methods for
re-using, but it is rather pointless to try. Just create a new one.
> Becuase it is storing the actual connection and session.
The connection is kept in the method so you can release it when you're
done with the response. The connection manager will try to keep the
connection alive for the next request, but that is strictly a performance
optimization.
The session is in no way related to a connection. Session cookies are stored
in the HttpState. If you didn't create an extra HttpState, the default state
in the HttpClient will be used.
I recommend that you create a new method object to access the second link,
and execute it with the same HttpClient object as the first one. Don't
forget to release the connection after each request, but not before you're
done with reading the response.
hope that helps,
Roland
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org