You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Mark Southern <so...@scripps.edu> on 2011/08/19 02:48:31 UTC
Truncated output with HttpPost but works with cURL
To whomever can help me!
I am submitting to a web service and receiving an xml file in return. I am using POST b/c this is too large for a GET. If I use cURL then this works. However with HttpClient (4.1.2) and the same parameters, I am getting a truncated xml file returned.
Does anyone have an idea what might be wrong? I include the test file but not the whole list of parameters I am submitting as this is a further 350Kb.
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.xml.sax.helpers.DefaultHandler;
public class HttpClientTest {
public static void main(String[] args) throws Exception {
new HttpClientTest().runTest();
}
public void runTest() throws Exception {
List<NameValuePair> list = new ArrayList(50000);
list.add(new BasicNameValuePair("db", "pcassay"));
list.add(new BasicNameValuePair("dbfrom", "pcassay"));
// read in ~50000 ids from file. These are existing uids in the pcassay
// database.
BufferedReader reader = new BufferedReader(new InputStreamReader(getClass().getClassLoader().getResourceAsStream("ids.txt")));
String line = null;
while (null != (line = reader.readLine())) {
list.add(new BasicNameValuePair("id", line));
}
list.add(new BasicNameValuePair("term", "summary[activityoutcomemethod]"));
list.add(new BasicNameValuePair("version", "2.0"));
list.add(new BasicNameValuePair("email", "...@scripps.edu"));
list.add(new BasicNameValuePair("tool", "...@scripps.edu"));
System.out.println("#NameValuePairs: " + list.size());
HttpPost post = new HttpPost("http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi");
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(list);
post.setEntity(entity);
HttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(post);
// dump to local file so we can refer to it.
File file = File.createTempFile("eutils", ".xml");
System.out.println("Copying eUtils stream to: " + file);
response.getEntity().writeTo(new BufferedOutputStream(new FileOutputStream(file)));
InputStream is = new BufferedInputStream(new FileInputStream(file));
// attempt to parse with sax. There will be an error if only a fragment
// of the xml is returned.
SAXParserFactory factory = SAXParserFactory.newInstance();
SAXParser saxParser = factory.newSAXParser();
saxParser.parse(is, new DefaultHandler() {
});
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Truncated output with HttpPost but works with cURL
Posted by Mark Southern <so...@scripps.edu>.
Hi Sam,
Thank you. That does seem definitive. I had tried several times with cURL yesterday that all completed but I never got it to work with HttpClient! I repeated it today and I see truncated files with cURL as well.
I'll follow up on the server side. Thanks again!
-----Original Message-----
From: Sam Crawford [mailto:samcrawford@gmail.com]
Sent: Friday, August 19, 2011 11:48 AM
To: HttpClient User Discussion
Subject: Re: Truncated output with HttpPost but works with cURL
Yes, it appears that the server is the cause. I'm not sure how familiar you are with packet captures, but I've shown the final four packets in the exchange at http://pastebin.com/EDE4KhZr
You'll see that www.ncbi.nlm.nih.gov sends a TCP FIN on line 5, and that the chunked response is ended within the same packet on line 57 (the "0" indicates the end of a chunked response). The client (curl, on 192.168.2.100) correctly responds with a FIN as well, and then the connection is closed.
The TCP FIN occurred almost exactly 7 minutes after the HTTP request first began. I strongly suspect that the server side script that's processing this data has a maximum execution time of 7 minutes set.
Hope this helps,
Sam
On 19 August 2011 16:32, Sam Crawford <sa...@gmail.com> wrote:
> Hrmmm, I've not run your Java code, but I think the server may be at
> fault here. I'm seeing the response being cut short in curl as well.
> It takes a very long time to complete the transaction, but I'm running
> it again with a packet capture right now so should be able to see what
> is taking place shortly.
>
>
> On 19 August 2011 15:58, Mark Southern <so...@scripps.edu> wrote:
>> Further details from the HttpResponse Entity:
>>
>> Response encoding: null
>> Response content length: -1
>> Chunked?: true
>> Content-Type: text/xml; charset=UTF-8
>>
>>
>> -----Original Message-----
>> From: kim young ill [mailto:khiemu@googlemail.com]
>> Sent: Friday, August 19, 2011 10:02 AM
>> To: HttpClient User Discussion
>> Subject: Re: Truncated output with HttpPost but works with cURL
>>
>> could be that response is gzipped without specifying content-encoding which leads to wrong content-length ?
>>
>> On Fri, Aug 19, 2011 at 3:32 PM, Mark Southern <so...@scripps.edu> wrote:
>>
>>> Dear Peter,
>>>
>>> Unfortunately that's not it. The problem occurs whether or not I
>>> include the step to dump to the local file (and also with an
>>> explicit flush/close added to the BufferedOutputStream).
>>>
>>> I have made available the java source, ids text file and cUrl
>>> example if case anyone cares for a further look:
>>>
>>> http://chemutils.florida.scripps.edu/httpclient/curl.sh
>>> http://chemutils.florida.scripps.edu/httpclient/ids.txt
>>> http://chemutils.florida.scripps.edu/httpclient/HttpClientTest.java
>>>
>>> Thank you,
>>> ~Mark.
>>>
>>> -----Original Message-----
>>> From: Pete Keyes [mailto:]
>>> Sent: Thursday, August 18, 2011 9:29 PM
>>> To: HttpClient User Discussion
>>> Subject: RE: Truncated output with HttpPost but works with cURL
>>>
>>> This is just a guess, but I don't see where you've closed the
>>> BufferedOutputStream passed to the writeTo() method.
>>>
>>> ...Pete Keyes
>>> Starbucks Coffee Co.
>>>
>>> ________________________________________
>>> From: Mark Southern [southern@scripps.edu]
>>> Sent: Thursday, August 18, 2011 05:48 PM
>>> To: httpclient-users@hc.apache.org
>>> Subject: Truncated output with HttpPost but works with cURL
>>>
>>> To whomever can help me!
>>>
>>> I am submitting to a web service and receiving an xml file in return.
>>> I am using POST b/c this is too large for a GET. If I use cURL then this works.
>>> However with HttpClient (4.1.2) and the same parameters, I am
>>> getting a truncated xml file returned.
>>>
>>> Does anyone have an idea what might be wrong? I include the test
>>> file but not the whole list of parameters I am submitting as this is a further 350Kb.
>>>
>>> import java.io.BufferedInputStream;
>>> import java.io.BufferedOutputStream; import java.io.BufferedReader;
>>> import java.io.File; import java.io.FileInputStream; import
>>> java.io.FileOutputStream; import java.io.InputStream; import
>>> java.io.InputStreamReader; import java.util.ArrayList; import
>>> java.util.List;
>>>
>>> import javax.xml.parsers.SAXParser;
>>> import javax.xml.parsers.SAXParserFactory;
>>>
>>> import org.apache.http.HttpResponse; import
>>> org.apache.http.NameValuePair; import
>>> org.apache.http.client.HttpClient;
>>> import org.apache.http.client.entity.UrlEncodedFormEntity;
>>> import org.apache.http.client.methods.HttpPost;
>>> import org.apache.http.impl.client.DefaultHttpClient;
>>> import org.apache.http.message.BasicNameValuePair;
>>> import org.xml.sax.helpers.DefaultHandler;
>>>
>>> public class HttpClientTest {
>>>
>>> public static void main(String[] args) throws Exception {
>>> new HttpClientTest().runTest();
>>> }
>>>
>>> public void runTest() throws Exception {
>>> List<NameValuePair> list = new ArrayList(50000);
>>> list.add(new BasicNameValuePair("db", "pcassay"));
>>> list.add(new BasicNameValuePair("dbfrom",
>>> "pcassay"));
>>> // read in ~50000 ids from file. These are existing
>>> uids in the pcassay
>>> // database.
>>> BufferedReader reader = new BufferedReader(new
>>> InputStreamReader(getClass().getClassLoader().getResourceAsStream("i
>>> ds.txt")));
>>> String line = null;
>>> while (null != (line = reader.readLine())) {
>>> list.add(new BasicNameValuePair("id", line));
>>> }
>>> list.add(new BasicNameValuePair("term",
>>> "summary[activityoutcomemethod]"));
>>> list.add(new BasicNameValuePair("version", "2.0"));
>>> list.add(new BasicNameValuePair("email",
>>> "...@scripps.edu "));
>>> list.add(new BasicNameValuePair("tool",
>>> "...@scripps.edu "));
>>>
>>> System.out.println("#NameValuePairs: " +
>>> list.size());
>>>
>>> HttpPost post = new HttpPost("
>>> http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi");
>>> UrlEncodedFormEntity entity = new
>>> UrlEncodedFormEntity(list);
>>> post.setEntity(entity);
>>> HttpClient client = new DefaultHttpClient();
>>> HttpResponse response = client.execute(post);
>>>
>>> // dump to local file so we can refer to it.
>>> File file = File.createTempFile("eutils", ".xml");
>>> System.out.println("Copying eUtils stream to: " +
>>> file);
>>> response.getEntity().writeTo(new
>>> BufferedOutputStream(new FileOutputStream(file)));
>>> InputStream is = new BufferedInputStream(new
>>> FileInputStream(file));
>>>
>>> // attempt to parse with sax. There will be an error
>>> if only a fragment
>>> // of the xml is returned.
>>> SAXParserFactory factory =
>>> SAXParserFactory.newInstance();
>>> SAXParser saxParser = factory.newSAXParser();
>>> saxParser.parse(is, new DefaultHandler() {
>>> });
>>> }
>>> }
>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Truncated output with HttpPost but works with cURL
Posted by Sam Crawford <sa...@gmail.com>.
Yes, it appears that the server is the cause. I'm not sure how
familiar you are with packet captures, but I've shown the final four
packets in the exchange at http://pastebin.com/EDE4KhZr
You'll see that www.ncbi.nlm.nih.gov sends a TCP FIN on line 5, and
that the chunked response is ended within the same packet on line 57
(the "0" indicates the end of a chunked response). The client (curl,
on 192.168.2.100) correctly responds with a FIN as well, and then the
connection is closed.
The TCP FIN occurred almost exactly 7 minutes after the HTTP request
first began. I strongly suspect that the server side script that's
processing this data has a maximum execution time of 7 minutes set.
Hope this helps,
Sam
On 19 August 2011 16:32, Sam Crawford <sa...@gmail.com> wrote:
> Hrmmm, I've not run your Java code, but I think the server may be at
> fault here. I'm seeing the response being cut short in curl as well.
> It takes a very long time to complete the transaction, but I'm running
> it again with a packet capture right now so should be able to see what
> is taking place shortly.
>
>
> On 19 August 2011 15:58, Mark Southern <so...@scripps.edu> wrote:
>> Further details from the HttpResponse Entity:
>>
>> Response encoding: null
>> Response content length: -1
>> Chunked?: true
>> Content-Type: text/xml; charset=UTF-8
>>
>>
>> -----Original Message-----
>> From: kim young ill [mailto:khiemu@googlemail.com]
>> Sent: Friday, August 19, 2011 10:02 AM
>> To: HttpClient User Discussion
>> Subject: Re: Truncated output with HttpPost but works with cURL
>>
>> could be that response is gzipped without specifying content-encoding which leads to wrong content-length ?
>>
>> On Fri, Aug 19, 2011 at 3:32 PM, Mark Southern <so...@scripps.edu> wrote:
>>
>>> Dear Peter,
>>>
>>> Unfortunately that's not it. The problem occurs whether or not I
>>> include the step to dump to the local file (and also with an explicit
>>> flush/close added to the BufferedOutputStream).
>>>
>>> I have made available the java source, ids text file and cUrl example
>>> if case anyone cares for a further look:
>>>
>>> http://chemutils.florida.scripps.edu/httpclient/curl.sh
>>> http://chemutils.florida.scripps.edu/httpclient/ids.txt
>>> http://chemutils.florida.scripps.edu/httpclient/HttpClientTest.java
>>>
>>> Thank you,
>>> ~Mark.
>>>
>>> -----Original Message-----
>>> From: Pete Keyes [mailto:]
>>> Sent: Thursday, August 18, 2011 9:29 PM
>>> To: HttpClient User Discussion
>>> Subject: RE: Truncated output with HttpPost but works with cURL
>>>
>>> This is just a guess, but I don't see where you've closed the
>>> BufferedOutputStream passed to the writeTo() method.
>>>
>>> ...Pete Keyes
>>> Starbucks Coffee Co.
>>>
>>> ________________________________________
>>> From: Mark Southern [southern@scripps.edu]
>>> Sent: Thursday, August 18, 2011 05:48 PM
>>> To: httpclient-users@hc.apache.org
>>> Subject: Truncated output with HttpPost but works with cURL
>>>
>>> To whomever can help me!
>>>
>>> I am submitting to a web service and receiving an xml file in return.
>>> I am using POST b/c this is too large for a GET. If I use cURL then this works.
>>> However with HttpClient (4.1.2) and the same parameters, I am getting
>>> a truncated xml file returned.
>>>
>>> Does anyone have an idea what might be wrong? I include the test file
>>> but not the whole list of parameters I am submitting as this is a further 350Kb.
>>>
>>> import java.io.BufferedInputStream;
>>> import java.io.BufferedOutputStream;
>>> import java.io.BufferedReader;
>>> import java.io.File;
>>> import java.io.FileInputStream;
>>> import java.io.FileOutputStream;
>>> import java.io.InputStream;
>>> import java.io.InputStreamReader;
>>> import java.util.ArrayList;
>>> import java.util.List;
>>>
>>> import javax.xml.parsers.SAXParser;
>>> import javax.xml.parsers.SAXParserFactory;
>>>
>>> import org.apache.http.HttpResponse;
>>> import org.apache.http.NameValuePair;
>>> import org.apache.http.client.HttpClient;
>>> import org.apache.http.client.entity.UrlEncodedFormEntity;
>>> import org.apache.http.client.methods.HttpPost;
>>> import org.apache.http.impl.client.DefaultHttpClient;
>>> import org.apache.http.message.BasicNameValuePair;
>>> import org.xml.sax.helpers.DefaultHandler;
>>>
>>> public class HttpClientTest {
>>>
>>> public static void main(String[] args) throws Exception {
>>> new HttpClientTest().runTest();
>>> }
>>>
>>> public void runTest() throws Exception {
>>> List<NameValuePair> list = new ArrayList(50000);
>>> list.add(new BasicNameValuePair("db", "pcassay"));
>>> list.add(new BasicNameValuePair("dbfrom", "pcassay"));
>>> // read in ~50000 ids from file. These are existing
>>> uids in the pcassay
>>> // database.
>>> BufferedReader reader = new BufferedReader(new
>>> InputStreamReader(getClass().getClassLoader().getResourceAsStream("ids.txt")));
>>> String line = null;
>>> while (null != (line = reader.readLine())) {
>>> list.add(new BasicNameValuePair("id", line));
>>> }
>>> list.add(new BasicNameValuePair("term",
>>> "summary[activityoutcomemethod]"));
>>> list.add(new BasicNameValuePair("version", "2.0"));
>>> list.add(new BasicNameValuePair("email",
>>> "...@scripps.edu "));
>>> list.add(new BasicNameValuePair("tool",
>>> "...@scripps.edu "));
>>>
>>> System.out.println("#NameValuePairs: " + list.size());
>>>
>>> HttpPost post = new HttpPost("
>>> http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi");
>>> UrlEncodedFormEntity entity = new
>>> UrlEncodedFormEntity(list);
>>> post.setEntity(entity);
>>> HttpClient client = new DefaultHttpClient();
>>> HttpResponse response = client.execute(post);
>>>
>>> // dump to local file so we can refer to it.
>>> File file = File.createTempFile("eutils", ".xml");
>>> System.out.println("Copying eUtils stream to: " + file);
>>> response.getEntity().writeTo(new
>>> BufferedOutputStream(new FileOutputStream(file)));
>>> InputStream is = new BufferedInputStream(new
>>> FileInputStream(file));
>>>
>>> // attempt to parse with sax. There will be an error if
>>> only a fragment
>>> // of the xml is returned.
>>> SAXParserFactory factory = SAXParserFactory.newInstance();
>>> SAXParser saxParser = factory.newSAXParser();
>>> saxParser.parse(is, new DefaultHandler() {
>>> });
>>> }
>>> }
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Truncated output with HttpPost but works with cURL
Posted by Sam Crawford <sa...@gmail.com>.
Hrmmm, I've not run your Java code, but I think the server may be at
fault here. I'm seeing the response being cut short in curl as well.
It takes a very long time to complete the transaction, but I'm running
it again with a packet capture right now so should be able to see what
is taking place shortly.
On 19 August 2011 15:58, Mark Southern <so...@scripps.edu> wrote:
> Further details from the HttpResponse Entity:
>
> Response encoding: null
> Response content length: -1
> Chunked?: true
> Content-Type: text/xml; charset=UTF-8
>
>
> -----Original Message-----
> From: kim young ill [mailto:khiemu@googlemail.com]
> Sent: Friday, August 19, 2011 10:02 AM
> To: HttpClient User Discussion
> Subject: Re: Truncated output with HttpPost but works with cURL
>
> could be that response is gzipped without specifying content-encoding which leads to wrong content-length ?
>
> On Fri, Aug 19, 2011 at 3:32 PM, Mark Southern <so...@scripps.edu> wrote:
>
>> Dear Peter,
>>
>> Unfortunately that's not it. The problem occurs whether or not I
>> include the step to dump to the local file (and also with an explicit
>> flush/close added to the BufferedOutputStream).
>>
>> I have made available the java source, ids text file and cUrl example
>> if case anyone cares for a further look:
>>
>> http://chemutils.florida.scripps.edu/httpclient/curl.sh
>> http://chemutils.florida.scripps.edu/httpclient/ids.txt
>> http://chemutils.florida.scripps.edu/httpclient/HttpClientTest.java
>>
>> Thank you,
>> ~Mark.
>>
>> -----Original Message-----
>> From: Pete Keyes [mailto:]
>> Sent: Thursday, August 18, 2011 9:29 PM
>> To: HttpClient User Discussion
>> Subject: RE: Truncated output with HttpPost but works with cURL
>>
>> This is just a guess, but I don't see where you've closed the
>> BufferedOutputStream passed to the writeTo() method.
>>
>> ...Pete Keyes
>> Starbucks Coffee Co.
>>
>> ________________________________________
>> From: Mark Southern [southern@scripps.edu]
>> Sent: Thursday, August 18, 2011 05:48 PM
>> To: httpclient-users@hc.apache.org
>> Subject: Truncated output with HttpPost but works with cURL
>>
>> To whomever can help me!
>>
>> I am submitting to a web service and receiving an xml file in return.
>> I am using POST b/c this is too large for a GET. If I use cURL then this works.
>> However with HttpClient (4.1.2) and the same parameters, I am getting
>> a truncated xml file returned.
>>
>> Does anyone have an idea what might be wrong? I include the test file
>> but not the whole list of parameters I am submitting as this is a further 350Kb.
>>
>> import java.io.BufferedInputStream;
>> import java.io.BufferedOutputStream;
>> import java.io.BufferedReader;
>> import java.io.File;
>> import java.io.FileInputStream;
>> import java.io.FileOutputStream;
>> import java.io.InputStream;
>> import java.io.InputStreamReader;
>> import java.util.ArrayList;
>> import java.util.List;
>>
>> import javax.xml.parsers.SAXParser;
>> import javax.xml.parsers.SAXParserFactory;
>>
>> import org.apache.http.HttpResponse;
>> import org.apache.http.NameValuePair;
>> import org.apache.http.client.HttpClient;
>> import org.apache.http.client.entity.UrlEncodedFormEntity;
>> import org.apache.http.client.methods.HttpPost;
>> import org.apache.http.impl.client.DefaultHttpClient;
>> import org.apache.http.message.BasicNameValuePair;
>> import org.xml.sax.helpers.DefaultHandler;
>>
>> public class HttpClientTest {
>>
>> public static void main(String[] args) throws Exception {
>> new HttpClientTest().runTest();
>> }
>>
>> public void runTest() throws Exception {
>> List<NameValuePair> list = new ArrayList(50000);
>> list.add(new BasicNameValuePair("db", "pcassay"));
>> list.add(new BasicNameValuePair("dbfrom", "pcassay"));
>> // read in ~50000 ids from file. These are existing
>> uids in the pcassay
>> // database.
>> BufferedReader reader = new BufferedReader(new
>> InputStreamReader(getClass().getClassLoader().getResourceAsStream("ids.txt")));
>> String line = null;
>> while (null != (line = reader.readLine())) {
>> list.add(new BasicNameValuePair("id", line));
>> }
>> list.add(new BasicNameValuePair("term",
>> "summary[activityoutcomemethod]"));
>> list.add(new BasicNameValuePair("version", "2.0"));
>> list.add(new BasicNameValuePair("email",
>> "...@scripps.edu "));
>> list.add(new BasicNameValuePair("tool",
>> "...@scripps.edu "));
>>
>> System.out.println("#NameValuePairs: " + list.size());
>>
>> HttpPost post = new HttpPost("
>> http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi");
>> UrlEncodedFormEntity entity = new
>> UrlEncodedFormEntity(list);
>> post.setEntity(entity);
>> HttpClient client = new DefaultHttpClient();
>> HttpResponse response = client.execute(post);
>>
>> // dump to local file so we can refer to it.
>> File file = File.createTempFile("eutils", ".xml");
>> System.out.println("Copying eUtils stream to: " + file);
>> response.getEntity().writeTo(new
>> BufferedOutputStream(new FileOutputStream(file)));
>> InputStream is = new BufferedInputStream(new
>> FileInputStream(file));
>>
>> // attempt to parse with sax. There will be an error if
>> only a fragment
>> // of the xml is returned.
>> SAXParserFactory factory = SAXParserFactory.newInstance();
>> SAXParser saxParser = factory.newSAXParser();
>> saxParser.parse(is, new DefaultHandler() {
>> });
>> }
>> }
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Truncated output with HttpPost but works with cURL
Posted by Mark Southern <so...@scripps.edu>.
Further details from the HttpResponse Entity:
Response encoding: null
Response content length: -1
Chunked?: true
Content-Type: text/xml; charset=UTF-8
-----Original Message-----
From: kim young ill [mailto:khiemu@googlemail.com]
Sent: Friday, August 19, 2011 10:02 AM
To: HttpClient User Discussion
Subject: Re: Truncated output with HttpPost but works with cURL
could be that response is gzipped without specifying content-encoding which leads to wrong content-length ?
On Fri, Aug 19, 2011 at 3:32 PM, Mark Southern <so...@scripps.edu> wrote:
> Dear Peter,
>
> Unfortunately that's not it. The problem occurs whether or not I
> include the step to dump to the local file (and also with an explicit
> flush/close added to the BufferedOutputStream).
>
> I have made available the java source, ids text file and cUrl example
> if case anyone cares for a further look:
>
> http://chemutils.florida.scripps.edu/httpclient/curl.sh
> http://chemutils.florida.scripps.edu/httpclient/ids.txt
> http://chemutils.florida.scripps.edu/httpclient/HttpClientTest.java
>
> Thank you,
> ~Mark.
>
> -----Original Message-----
> From: Pete Keyes [mailto:]
> Sent: Thursday, August 18, 2011 9:29 PM
> To: HttpClient User Discussion
> Subject: RE: Truncated output with HttpPost but works with cURL
>
> This is just a guess, but I don't see where you've closed the
> BufferedOutputStream passed to the writeTo() method.
>
> ...Pete Keyes
> Starbucks Coffee Co.
>
> ________________________________________
> From: Mark Southern [southern@scripps.edu]
> Sent: Thursday, August 18, 2011 05:48 PM
> To: httpclient-users@hc.apache.org
> Subject: Truncated output with HttpPost but works with cURL
>
> To whomever can help me!
>
> I am submitting to a web service and receiving an xml file in return.
> I am using POST b/c this is too large for a GET. If I use cURL then this works.
> However with HttpClient (4.1.2) and the same parameters, I am getting
> a truncated xml file returned.
>
> Does anyone have an idea what might be wrong? I include the test file
> but not the whole list of parameters I am submitting as this is a further 350Kb.
>
> import java.io.BufferedInputStream;
> import java.io.BufferedOutputStream;
> import java.io.BufferedReader;
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.FileOutputStream;
> import java.io.InputStream;
> import java.io.InputStreamReader;
> import java.util.ArrayList;
> import java.util.List;
>
> import javax.xml.parsers.SAXParser;
> import javax.xml.parsers.SAXParserFactory;
>
> import org.apache.http.HttpResponse;
> import org.apache.http.NameValuePair;
> import org.apache.http.client.HttpClient;
> import org.apache.http.client.entity.UrlEncodedFormEntity;
> import org.apache.http.client.methods.HttpPost;
> import org.apache.http.impl.client.DefaultHttpClient;
> import org.apache.http.message.BasicNameValuePair;
> import org.xml.sax.helpers.DefaultHandler;
>
> public class HttpClientTest {
>
> public static void main(String[] args) throws Exception {
> new HttpClientTest().runTest();
> }
>
> public void runTest() throws Exception {
> List<NameValuePair> list = new ArrayList(50000);
> list.add(new BasicNameValuePair("db", "pcassay"));
> list.add(new BasicNameValuePair("dbfrom", "pcassay"));
> // read in ~50000 ids from file. These are existing
> uids in the pcassay
> // database.
> BufferedReader reader = new BufferedReader(new
> InputStreamReader(getClass().getClassLoader().getResourceAsStream("ids.txt")));
> String line = null;
> while (null != (line = reader.readLine())) {
> list.add(new BasicNameValuePair("id", line));
> }
> list.add(new BasicNameValuePair("term",
> "summary[activityoutcomemethod]"));
> list.add(new BasicNameValuePair("version", "2.0"));
> list.add(new BasicNameValuePair("email",
> "...@scripps.edu "));
> list.add(new BasicNameValuePair("tool",
> "...@scripps.edu "));
>
> System.out.println("#NameValuePairs: " + list.size());
>
> HttpPost post = new HttpPost("
> http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi");
> UrlEncodedFormEntity entity = new
> UrlEncodedFormEntity(list);
> post.setEntity(entity);
> HttpClient client = new DefaultHttpClient();
> HttpResponse response = client.execute(post);
>
> // dump to local file so we can refer to it.
> File file = File.createTempFile("eutils", ".xml");
> System.out.println("Copying eUtils stream to: " + file);
> response.getEntity().writeTo(new
> BufferedOutputStream(new FileOutputStream(file)));
> InputStream is = new BufferedInputStream(new
> FileInputStream(file));
>
> // attempt to parse with sax. There will be an error if
> only a fragment
> // of the xml is returned.
> SAXParserFactory factory = SAXParserFactory.newInstance();
> SAXParser saxParser = factory.newSAXParser();
> saxParser.parse(is, new DefaultHandler() {
> });
> }
> }
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Truncated output with HttpPost but works with cURL
Posted by kim young ill <kh...@googlemail.com>.
could be that response is gzipped without specifying content-encoding which
leads to wrong content-length ?
On Fri, Aug 19, 2011 at 3:32 PM, Mark Southern <so...@scripps.edu> wrote:
> Dear Peter,
>
> Unfortunately that's not it. The problem occurs whether or not I include
> the step to dump to the local file (and also with an explicit flush/close
> added to the BufferedOutputStream).
>
> I have made available the java source, ids text file and cUrl example if
> case anyone cares for a further look:
>
> http://chemutils.florida.scripps.edu/httpclient/curl.sh
> http://chemutils.florida.scripps.edu/httpclient/ids.txt
> http://chemutils.florida.scripps.edu/httpclient/HttpClientTest.java
>
> Thank you,
> ~Mark.
>
> -----Original Message-----
> From: Pete Keyes [mailto:]
> Sent: Thursday, August 18, 2011 9:29 PM
> To: HttpClient User Discussion
> Subject: RE: Truncated output with HttpPost but works with cURL
>
> This is just a guess, but I don't see where you've closed the
> BufferedOutputStream passed to the writeTo() method.
>
> ...Pete Keyes
> Starbucks Coffee Co.
>
> ________________________________________
> From: Mark Southern [southern@scripps.edu]
> Sent: Thursday, August 18, 2011 05:48 PM
> To: httpclient-users@hc.apache.org
> Subject: Truncated output with HttpPost but works with cURL
>
> To whomever can help me!
>
> I am submitting to a web service and receiving an xml file in return. I am
> using POST b/c this is too large for a GET. If I use cURL then this works.
> However with HttpClient (4.1.2) and the same parameters, I am getting a
> truncated xml file returned.
>
> Does anyone have an idea what might be wrong? I include the test file but
> not the whole list of parameters I am submitting as this is a further 350Kb.
>
> import java.io.BufferedInputStream;
> import java.io.BufferedOutputStream;
> import java.io.BufferedReader;
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.FileOutputStream;
> import java.io.InputStream;
> import java.io.InputStreamReader;
> import java.util.ArrayList;
> import java.util.List;
>
> import javax.xml.parsers.SAXParser;
> import javax.xml.parsers.SAXParserFactory;
>
> import org.apache.http.HttpResponse;
> import org.apache.http.NameValuePair;
> import org.apache.http.client.HttpClient;
> import org.apache.http.client.entity.UrlEncodedFormEntity;
> import org.apache.http.client.methods.HttpPost;
> import org.apache.http.impl.client.DefaultHttpClient;
> import org.apache.http.message.BasicNameValuePair;
> import org.xml.sax.helpers.DefaultHandler;
>
> public class HttpClientTest {
>
> public static void main(String[] args) throws Exception {
> new HttpClientTest().runTest();
> }
>
> public void runTest() throws Exception {
> List<NameValuePair> list = new ArrayList(50000);
> list.add(new BasicNameValuePair("db", "pcassay"));
> list.add(new BasicNameValuePair("dbfrom", "pcassay"));
> // read in ~50000 ids from file. These are existing uids in
> the pcassay
> // database.
> BufferedReader reader = new BufferedReader(new
> InputStreamReader(getClass().getClassLoader().getResourceAsStream("ids.txt")));
> String line = null;
> while (null != (line = reader.readLine())) {
> list.add(new BasicNameValuePair("id", line));
> }
> list.add(new BasicNameValuePair("term",
> "summary[activityoutcomemethod]"));
> list.add(new BasicNameValuePair("version", "2.0"));
> list.add(new BasicNameValuePair("email", "...@scripps.edu
> "));
> list.add(new BasicNameValuePair("tool", "...@scripps.edu
> "));
>
> System.out.println("#NameValuePairs: " + list.size());
>
> HttpPost post = new HttpPost("
> http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi");
> UrlEncodedFormEntity entity = new
> UrlEncodedFormEntity(list);
> post.setEntity(entity);
> HttpClient client = new DefaultHttpClient();
> HttpResponse response = client.execute(post);
>
> // dump to local file so we can refer to it.
> File file = File.createTempFile("eutils", ".xml");
> System.out.println("Copying eUtils stream to: " + file);
> response.getEntity().writeTo(new BufferedOutputStream(new
> FileOutputStream(file)));
> InputStream is = new BufferedInputStream(new
> FileInputStream(file));
>
> // attempt to parse with sax. There will be an error if only
> a fragment
> // of the xml is returned.
> SAXParserFactory factory = SAXParserFactory.newInstance();
> SAXParser saxParser = factory.newSAXParser();
> saxParser.parse(is, new DefaultHandler() {
> });
> }
> }
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
RE: Truncated output with HttpPost but works with cURL
Posted by Mark Southern <so...@scripps.edu>.
Dear Peter,
Unfortunately that's not it. The problem occurs whether or not I include the step to dump to the local file (and also with an explicit flush/close added to the BufferedOutputStream).
I have made available the java source, ids text file and cUrl example if case anyone cares for a further look:
http://chemutils.florida.scripps.edu/httpclient/curl.sh
http://chemutils.florida.scripps.edu/httpclient/ids.txt
http://chemutils.florida.scripps.edu/httpclient/HttpClientTest.java
Thank you,
~Mark.
-----Original Message-----
From: Pete Keyes [mailto:]
Sent: Thursday, August 18, 2011 9:29 PM
To: HttpClient User Discussion
Subject: RE: Truncated output with HttpPost but works with cURL
This is just a guess, but I don't see where you've closed the BufferedOutputStream passed to the writeTo() method.
...Pete Keyes
Starbucks Coffee Co.
________________________________________
From: Mark Southern [southern@scripps.edu]
Sent: Thursday, August 18, 2011 05:48 PM
To: httpclient-users@hc.apache.org
Subject: Truncated output with HttpPost but works with cURL
To whomever can help me!
I am submitting to a web service and receiving an xml file in return. I am using POST b/c this is too large for a GET. If I use cURL then this works. However with HttpClient (4.1.2) and the same parameters, I am getting a truncated xml file returned.
Does anyone have an idea what might be wrong? I include the test file but not the whole list of parameters I am submitting as this is a further 350Kb.
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.xml.sax.helpers.DefaultHandler;
public class HttpClientTest {
public static void main(String[] args) throws Exception {
new HttpClientTest().runTest();
}
public void runTest() throws Exception {
List<NameValuePair> list = new ArrayList(50000);
list.add(new BasicNameValuePair("db", "pcassay"));
list.add(new BasicNameValuePair("dbfrom", "pcassay"));
// read in ~50000 ids from file. These are existing uids in the pcassay
// database.
BufferedReader reader = new BufferedReader(new InputStreamReader(getClass().getClassLoader().getResourceAsStream("ids.txt")));
String line = null;
while (null != (line = reader.readLine())) {
list.add(new BasicNameValuePair("id", line));
}
list.add(new BasicNameValuePair("term", "summary[activityoutcomemethod]"));
list.add(new BasicNameValuePair("version", "2.0"));
list.add(new BasicNameValuePair("email", "...@scripps.edu"));
list.add(new BasicNameValuePair("tool", "...@scripps.edu"));
System.out.println("#NameValuePairs: " + list.size());
HttpPost post = new HttpPost("http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi");
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(list);
post.setEntity(entity);
HttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(post);
// dump to local file so we can refer to it.
File file = File.createTempFile("eutils", ".xml");
System.out.println("Copying eUtils stream to: " + file);
response.getEntity().writeTo(new BufferedOutputStream(new FileOutputStream(file)));
InputStream is = new BufferedInputStream(new FileInputStream(file));
// attempt to parse with sax. There will be an error if only a fragment
// of the xml is returned.
SAXParserFactory factory = SAXParserFactory.newInstance();
SAXParser saxParser = factory.newSAXParser();
saxParser.parse(is, new DefaultHandler() {
});
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Truncated output with HttpPost but works with cURL
Posted by Pete Keyes <PK...@starbucks.com>.
This is just a guess, but I don't see where you've closed the BufferedOutputStream passed to the writeTo() method.
...Pete Keyes
Starbucks Coffee Co.
________________________________________
From: Mark Southern [southern@scripps.edu]
Sent: Thursday, August 18, 2011 05:48 PM
To: httpclient-users@hc.apache.org
Subject: Truncated output with HttpPost but works with cURL
To whomever can help me!
I am submitting to a web service and receiving an xml file in return. I am using POST b/c this is too large for a GET. If I use cURL then this works. However with HttpClient (4.1.2) and the same parameters, I am getting a truncated xml file returned.
Does anyone have an idea what might be wrong? I include the test file but not the whole list of parameters I am submitting as this is a further 350Kb.
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.xml.sax.helpers.DefaultHandler;
public class HttpClientTest {
public static void main(String[] args) throws Exception {
new HttpClientTest().runTest();
}
public void runTest() throws Exception {
List<NameValuePair> list = new ArrayList(50000);
list.add(new BasicNameValuePair("db", "pcassay"));
list.add(new BasicNameValuePair("dbfrom", "pcassay"));
// read in ~50000 ids from file. These are existing uids in the pcassay
// database.
BufferedReader reader = new BufferedReader(new InputStreamReader(getClass().getClassLoader().getResourceAsStream("ids.txt")));
String line = null;
while (null != (line = reader.readLine())) {
list.add(new BasicNameValuePair("id", line));
}
list.add(new BasicNameValuePair("term", "summary[activityoutcomemethod]"));
list.add(new BasicNameValuePair("version", "2.0"));
list.add(new BasicNameValuePair("email", "...@scripps.edu"));
list.add(new BasicNameValuePair("tool", "...@scripps.edu"));
System.out.println("#NameValuePairs: " + list.size());
HttpPost post = new HttpPost("http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi");
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(list);
post.setEntity(entity);
HttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(post);
// dump to local file so we can refer to it.
File file = File.createTempFile("eutils", ".xml");
System.out.println("Copying eUtils stream to: " + file);
response.getEntity().writeTo(new BufferedOutputStream(new FileOutputStream(file)));
InputStream is = new BufferedInputStream(new FileInputStream(file));
// attempt to parse with sax. There will be an error if only a fragment
// of the xml is returned.
SAXParserFactory factory = SAXParserFactory.newInstance();
SAXParser saxParser = factory.newSAXParser();
saxParser.parse(is, new DefaultHandler() {
});
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org