You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by dahoffer <dh...@gmail.com> on 2008/10/28 01:47:54 UTC
How to download from URL that requires login?
I'm new to HTTP Client. I have a URL I want to connect to (and download
content) that requires user to first login with username & password. How
can I do this with HTTP Client?
-Dave
--
View this message in context: http://www.nabble.com/How-to-download-from-URL-that-requires-login--tp20199503p20199503.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: How to download from URL that requires login?
Posted by Florent Blondeau <fb...@pingwy.com>.
Hi dave,
I understand you want to access a login page, login then download the
content of the accessed web page
HttpClient is a great tool, but if you want to do what you describe, a
tool like Webtest may fit your needs (webtest.canoo.com) especially
with its groovy support.
If you really want to use HttpClient, you'll have to handle the session
proposed by the server by your own.
If it's just an HTTP authentication , an url like
http://username:password@mysite.com/loginpage.html should be sufficient
to get direct login, am I wrong ?
florent blondeau
Pingwy
27, rue des arènes
49100 Angers
dahoffer a écrit :
> I'm new to HTTP Client. I have a URL I want to connect to (and download
> content) that requires user to first login with username & password. How
> can I do this with HTTP Client?
>
> -Dave
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: How to download from URL that requires login?
Posted by AnuRahim <an...@gmail.com>.
Hi , I have tried this code snippt I am getting the following error
Exception in thread "main" java.lang.RuntimeException: Stub!
at org.apache.http.conn.scheme.SchemeRegistry.<init>(SchemeRegistry.java:4)
at
org.vietspider.net.client.HTTPClientFactory.createRegitry(HTTPClientFactory.java:95)
at
org.vietspider.net.client.HTTPClientFactory.createDefaultHttpClient(HTTPClientFactory.java:82)
at org.vietspider.net.client.WebClient.<init>(WebClient.java:90)
at com.calpine.android.YahooLogin.main(YahooLogin.java:27)
can any one please tel me where I went wrong?
Thuannd wrote:
>
> use vietspider htmlparser build 8,
> download
> http://downloads.sourceforge.net/binhgiang/HTMLParser2_Build8.zip?use_mirror=
>
> Example: Code login Yahoo Mail.
>
> import java.io.File;
> import java.net.URL;
> import java.util.List;
>
> import org.apache.http.HttpEntity;
> import org.apache.http.HttpHost;
> import org.apache.http.HttpResponse;
> import org.apache.http.client.methods.HttpGet;
> import org.apache.http.cookie.Cookie;
> import org.apache.http.impl.client.DefaultHttpClient;
> import org.vietspider.browser.HttpSessionUtils;
> import org.vietspider.common.io.DataWriter;
> import org.vietspider.net.client.HttpResponseReader;
> import org.vietspider.net.client.WebClient;
>
> /**
> * Author : Nhu Dinh Thuan
> * nhudinhthuan@yahoo.com
> * Nov 2, 2008
> */
> public class YahooLogin {
>
> public static void main(String[] args) throws Exception {
> WebClient webClient = new WebClient();
>
> String homepage = "http://mail.yahoo.com/";
> webClient.setURL(homepage, new URL(homepage));
>
> HttpHost httpHost = webClient.createHttpHost(homepage);
> HttpGet httpGet = webClient.createGetMethod(homepage,
> "http://www.yahoo.com");
>
> HttpResponse response = webClient.execute(httpHost, httpGet);
> HttpEntity entity = response.getEntity();
>
> System.out.println("Login form get: " + response.getStatusLine());
> if (entity != null) entity.consumeContent();
>
> System.out.println("Initial set of cookies:");
> DefaultHttpClient httpClient = (DefaultHttpClient)
> webClient.getHttpClient();
> List<Cookie> cookies = httpClient.getCookieStore().getCookies();
> if (cookies.isEmpty()) {
> System.out.println("None");
> } else {
> for (int i = 0; i < cookies.size(); i++) {
> System.out.println("- " + cookies.get(i).toString());
> }
> }
>
> HttpSessionUtils httpSession = new HttpSessionUtils(webClient,
> "ERROR");
>
> StringBuilder builder = new StringBuilder(homepage);
> builder.append('\n').append("username:password");
>
> httpSession.login(builder.toString(), "utf-8", new URL(homepage),
> homepage);
>
> httpGet = webClient.createGetMethod("http://my.yahoo.com",
> "http://www.yahoo.com");
> response = webClient.execute(httpHost, httpGet);
> entity = response.getEntity();
>
>
> HttpResponseReader httpResponseReader = new HttpResponseReader();
> byte [] bytes = httpResponseReader.readBody(response);
> new DataWriter().save(new File("my_yahoo.html"), bytes);
>
> System.out.println("Login form get: " + response.getStatusLine());
> if (entity != null) entity.consumeContent();
>
> System.out.println("Post logon cookies:");
> cookies = httpClient.getCookieStore().getCookies();
> if (cookies.isEmpty()) {
> System.out.println("None");
> } else {
> for (int i = 0; i < cookies.size(); i++) {
> System.out.println("- " + cookies.get(i).toString());
> }
> }
>
> }
> }
>
>
> Code login Gmail:
>
>
> HttpSessionUtils httpSession = new HttpSessionUtils(webClient,
> "ERROR");
>
> StringBuilder builder = new
> StringBuilder("https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=1k96igf4806cy<mpl=default<mplcache=2");
> builder.append('\n').append("username:password");
>
> httpSession.login(builder.toString(), "utf-8", new URL(homepage),
> homepage);
>
> Code login to java.net:
>
> String homepage = "http://java.net/";
> HttpSessionUtils httpSession = new HttpSessionUtils(webClient,
> "ERROR");
> StringBuilder builder = new StringBuilder(homepage);
> builder.append('\n').append("username:password");
> httpSession.login(builder.toString(), "utf-8", new URL(homepage),
> homepage);
>
>
>
> See more example folder in HTMLParser_Build8
>
>
>
>
--
View this message in context: http://www.nabble.com/How-to-download-from-URL-that-requires-login--tp20199503p20361360.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: How to download from URL that requires login?
Posted by Thuannd <nh...@yahoo.com>.
use vietspider htmlparser build 8,
download
http://downloads.sourceforge.net/binhgiang/HTMLParser2_Build8.zip?use_mirror=
Example: Code login Yahoo Mail.
import java.io.File;
import java.net.URL;
import java.util.List;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.cookie.Cookie;
import org.apache.http.impl.client.DefaultHttpClient;
import org.vietspider.browser.HttpSessionUtils;
import org.vietspider.common.io.DataWriter;
import org.vietspider.net.client.HttpResponseReader;
import org.vietspider.net.client.WebClient;
/**
* Author : Nhu Dinh Thuan
* nhudinhthuan@yahoo.com
* Nov 2, 2008
*/
public class YahooLogin {
public static void main(String[] args) throws Exception {
WebClient webClient = new WebClient();
String homepage = "http://mail.yahoo.com/";
webClient.setURL(homepage, new URL(homepage));
HttpHost httpHost = webClient.createHttpHost(homepage);
HttpGet httpGet = webClient.createGetMethod(homepage,
"http://www.yahoo.com");
HttpResponse response = webClient.execute(httpHost, httpGet);
HttpEntity entity = response.getEntity();
System.out.println("Login form get: " + response.getStatusLine());
if (entity != null) entity.consumeContent();
System.out.println("Initial set of cookies:");
DefaultHttpClient httpClient = (DefaultHttpClient)
webClient.getHttpClient();
List<Cookie> cookies = httpClient.getCookieStore().getCookies();
if (cookies.isEmpty()) {
System.out.println("None");
} else {
for (int i = 0; i < cookies.size(); i++) {
System.out.println("- " + cookies.get(i).toString());
}
}
HttpSessionUtils httpSession = new HttpSessionUtils(webClient, "ERROR");
StringBuilder builder = new StringBuilder(homepage);
builder.append('\n').append("username:password");
httpSession.login(builder.toString(), "utf-8", new URL(homepage),
homepage);
httpGet = webClient.createGetMethod("http://my.yahoo.com",
"http://www.yahoo.com");
response = webClient.execute(httpHost, httpGet);
entity = response.getEntity();
HttpResponseReader httpResponseReader = new HttpResponseReader();
byte [] bytes = httpResponseReader.readBody(response);
new DataWriter().save(new File("my_yahoo.html"), bytes);
System.out.println("Login form get: " + response.getStatusLine());
if (entity != null) entity.consumeContent();
System.out.println("Post logon cookies:");
cookies = httpClient.getCookieStore().getCookies();
if (cookies.isEmpty()) {
System.out.println("None");
} else {
for (int i = 0; i < cookies.size(); i++) {
System.out.println("- " + cookies.get(i).toString());
}
}
}
}
Code login Gmail:
HttpSessionUtils httpSession = new HttpSessionUtils(webClient, "ERROR");
StringBuilder builder = new
StringBuilder("https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=1k96igf4806cy<mpl=default<mplcache=2");
builder.append('\n').append("username:password");
httpSession.login(builder.toString(), "utf-8", new URL(homepage),
homepage);
Code login to java.net:
String homepage = "http://java.net/";
HttpSessionUtils httpSession = new HttpSessionUtils(webClient,
"ERROR");
StringBuilder builder = new StringBuilder(homepage);
builder.append('\n').append("username:password");
httpSession.login(builder.toString(), "utf-8", new URL(homepage),
homepage);
See more example folder in HTMLParser_Build8
--
View this message in context: http://www.nabble.com/How-to-download-from-URL-that-requires-login--tp20199503p20288466.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: How to download from URL that requires login?
Posted by David Hoffer <da...@issinc.com>.
Christine, thanks much the URL is helpful.
-Dave
-----Original Message-----
From: Christine [mailto:christine@christine.nl]
Sent: Tuesday, October 28, 2008 5:05 AM
To: HttpClient User Discussion
Subject: Re: How to download from URL that requires login?
On Mon, 2008-10-27 at 17:47 -0700, dahoffer wrote:
> I'm new to HTTP Client. I have a URL I want to connect to (and download
> content) that requires user to first login with username & password. How
> can I do this with HTTP Client?
Dave,
httpClient provides support for posting and getting http content. What's
in the content, is irrelevant to httpClient. Logging into a web site
means you have to study the content of the login page, then create the
proper http requests to make your login. Apache has a very good tutorial
for this:
http://wiki.apache.org/jakarta-httpclient/ForAbsoluteBeginners
With this tutorial, I'm sure you can make your own login procedure. It's
not rocket science.
dagdag
Christine
> -Dave
--
dagdag is just a two character rotation of byebye
www.christine.nl
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: How to download from URL that requires login?
Posted by Christine <ch...@christine.nl>.
On Mon, 2008-10-27 at 17:47 -0700, dahoffer wrote:
> I'm new to HTTP Client. I have a URL I want to connect to (and download
> content) that requires user to first login with username & password. How
> can I do this with HTTP Client?
Dave,
httpClient provides support for posting and getting http content. What's
in the content, is irrelevant to httpClient. Logging into a web site
means you have to study the content of the login page, then create the
proper http requests to make your login. Apache has a very good tutorial
for this:
http://wiki.apache.org/jakarta-httpclient/ForAbsoluteBeginners
With this tutorial, I'm sure you can make your own login procedure. It's
not rocket science.
dagdag
Christine
> -Dave
--
dagdag is just a two character rotation of byebye
www.christine.nl
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org