You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by "Cesar, Scott" <Sc...@intuit.com.INVALID> on 2020/10/06 02:20:38 UTC
Converting SimpleHttpRequest to ClassicHttpRequest and Producing
SimpleHttpResponses
Hi,
I'm working to build a streaming HTTP library for Apache Beam. As part of providing a low entry effort
interface we want to provide asynchronous and synchronous options which rely on
SimpleHttpRequest / SimpleHttpResponse.
As the current synchronous client only handles ClassicHttpRequests, I'm currently using:
public static ClassicHttpRequest convert(SimpleHttpRequest original){
BasicClassicHttpRequest copy = new BasicClassicHttpRequest(original.getMethod(), original.getRequestUri());
copy.setVersion(original.getVersion());
for (final Iterator<Header> it = original.headerIterator(); it.hasNext(); ) {
copy.addHeader(it.next());
}
copy.setScheme(original.getScheme());
copy.setAuthority(original.getAuthority());
HttpEntity entity;
SimpleBody body = original.getBody();
if(body != null){
if (body.isBytes()){
entity = new ByteArrayEntity(body.getBodyBytes(), body.getContentType());
} else{
entity = new StringEntity(body.getBodyText(), body.getContentType());
}
copy.setEntity(entity);
}
return copy;
}
To convert simple requests into classic ones. Is this the best way to go about it (and if so, would this be
welcome as a pull request on SimpleHttpRequest), or is there a better way to provide this support (e.g.
extending the classic client to be able to handle SimpleRequests directly)
Similarly, I'm using a relatively involved method of greedily consuming a ClassicHttpResponse to produce a
SimpleHttpResponse from the classic client (code included at the end), and am curious if there's a better way
of doing this conversion or, if not, if providing a SimpleHttpResponseHandler and making it available from the client would be a good way of providing a unified interface going forewards.
public class SimpleHttpResponseHandler implements HttpClientResponseHandler<SimpleHttpResponse> {
@Override
public SimpleHttpResponse handleResponse(ClassicHttpResponse response) throws HttpException, IOException {
SimpleHttpResponse simple = SimpleHttpResponse.copy(response);
HttpEntity ent = response.getEntity();
if(ent != null){
byte[] data;
if (ent.getContentLength() >0) {
if(ent.getContentLength() > Integer.MAX_VALUE){
throw new RuntimeException("Received to much data to buffer in memory, try using a streaming consumer");
}
data = new byte[(int) ent.getContentLength()];
int read = ent.getContent().read(data);
if(read != ent.getContentLength())
throw new RuntimeException("Received "+read+" bytes instead of "+ent.getContentLength());
} else{
final int bufferSize = 8192;
InputStream is = ent.getContent();
ArrayList<byte[]> buffers = new ArrayList<>();
byte[] swapBuf = new byte[bufferSize];
int read = is.read(swapBuf);
while(read != -1 && read == bufferSize){
buffers.add(swapBuf);
swapBuf = new byte[bufferSize];
read = is.read(swapBuf);
}
int content = bufferSize*buffers.size();
if(read != -1)
content += read;
if(content <= 0)
throw new RuntimeException("Received to many bytes to buffer in memory");
data = new byte[content];
int index = 0;
for (final byte[] toCpy : buffers){
System.arraycopy(toCpy, 0, data, bufferSize*index, bufferSize);
index +=1;
}
if(read != -1)
System.arraycopy(swapBuf, 0, data, bufferSize*index, bufferSize);
swapBuf = null;
buffers.clear();
}
simple.setBody(
data,
ContentType.create(ent.getContentType())
);
}
return simple;
}
}
Thank you for your time,
Scott
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Converting SimpleHttpRequest to ClassicHttpRequest and
Producing SimpleHttpResponses
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2020-10-06 at 20:46 +0000, Cesar, Scott wrote:
> > >
...
> > On Tue, 2020-10-06 at 02:20 +0000, Cesar, Scott wrote:
> > Hi Scott
> >
> > Your code looks correct. However I am wondering why one would want
> > to
> > convert streaming messages into simple ones in the first place.
> >
> > Oleg
>
> Hi Oleg,
> There are two reasons I'm interested in turning the streaming
> responses from the classic client into simple messages:
>
> 1) Allowing users to provide very simple in memory parsing solutions
> which will run against both synchronous and asynchronous modules.
>
> 2) Beam has very restrictive serializability requirements. This means
> not all out of the box streaming consumption tools available can be
> put to work, at least not without some syntactic tradeoffs, so
> providing users simple and involved platforms to build off of seems
> to make the most sense.
>
> Scott
>
Scott
I do not know much (anything in fact) about Apache Beam but the whole
idea of storing message content in memory just to have the users parse
it into some higher level domain object sounds a bit, well,
questionable.
Otherwise your code seems OK.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Converting SimpleHttpRequest to ClassicHttpRequest and Producing
SimpleHttpResponses
Posted by "Cesar, Scott" <Sc...@intuit.com.INVALID>.
>On Tue, 2020-10-06 at 02:20 +0000, Cesar, Scott wrote:
>> Hi,
>> I'm working to build a streaming HTTP library for Apache Beam. As
>> part of providing a low entry effort
>> interface we want to provide asynchronous and synchronous options
>> which rely on
>> SimpleHttpRequest / SimpleHttpResponse.
>>
>> As the current synchronous client only handles ClassicHttpRequests,
>> I'm currently using:
>> public static ClassicHttpRequest convert(SimpleHttpRequest original){
>> BasicClassicHttpRequest copy = new
>> BasicClassicHttpRequest(original.getMethod(),
>> original.getRequestUri());
>> copy.setVersion(original.getVersion());
>> for (final Iterator<Header> it = original.headerIterator();
>> it.hasNext(); ) {
>> copy.addHeader(it.next());
>> }
>> copy.setScheme(original.getScheme());
>> copy.setAuthority(original.getAuthority());
>>
>> HttpEntity entity;
>> SimpleBody body = original.getBody();
>> if(body != null){
>> if (body.isBytes()){
>> entity = new ByteArrayEntity(body.getBodyBytes(),
>> body.getContentType());
>> } else{
>> entity = new StringEntity(body.getBodyText(),
>> body.getContentType());
>> }
>> copy.setEntity(entity);
>> }
>> return copy;
>> }
>>
>> To convert simple requests into classic ones. Is this the best way to
>> go about it (and if so, would this be
>> welcome as a pull request on SimpleHttpRequest), or is there a better
>> way to provide this support (e.g.
>> extending the classic client to be able to handle SimpleRequests
>> directly)
>>
>> Similarly, I'm using a relatively involved method of greedily
>> consuming a ClassicHttpResponse to produce a
>> SimpleHttpResponse from the classic client (code included at the
>> end), and am curious if there's a better way
>> of doing this conversion or, if not, if providing a
>> SimpleHttpResponseHandler and making it available from the client
>> would be a good way of providing a unified interface going forewards.
>>
>> public class SimpleHttpResponseHandler implements
>> HttpClientResponseHandler<SimpleHttpResponse> {
>>
>> @Override
>> public SimpleHttpResponse handleResponse(ClassicHttpResponse
>> response) throws HttpException, IOException {
>> SimpleHttpResponse simple =
>> SimpleHttpResponse.copy(response);
>> HttpEntity ent = response.getEntity();
>> if(ent != null){
>> byte[] data;
>> if (ent.getContentLength() >0) {
>> if(ent.getContentLength() > Integer.MAX_VALUE){
>> throw new RuntimeException("Received to much data
>> to buffer in memory, try using a streaming consumer");
>> }
>> data = new byte[(int) ent.getContentLength()];
>> int read = ent.getContent().read(data);
>> if(read != ent.getContentLength())
>> throw new RuntimeException("Received "+read+"
>> bytes instead of "+ent.getContentLength());
>> } else{
>> final int bufferSize = 8192;
>> InputStream is = ent.getContent();
>> ArrayList<byte[]> buffers = new ArrayList<>();
>> byte[] swapBuf = new byte[bufferSize];
>> int read = is.read(swapBuf);
>> while(read != -1 && read == bufferSize){
>> buffers.add(swapBuf);
>> swapBuf = new byte[bufferSize];
>> read = is.read(swapBuf);
>> }
>>
>> int content = bufferSize*buffers.size();
>> if(read != -1)
>> content += read;
>>
>> if(content <= 0)
>> throw new RuntimeException("Received to many
>> bytes to buffer in memory");
>> data = new byte[content];
>>
>> int index = 0;
>> for (final byte[] toCpy : buffers){
>> System.arraycopy(toCpy, 0, data,
>> bufferSize*index, bufferSize);
>> index +=1;
>> }
>> if(read != -1)
>> System.arraycopy(swapBuf, 0, data,
>> bufferSize*index, bufferSize);
>>
>> swapBuf = null;
>> buffers.clear();
>> }
>>
>> simple.setBody(
>> data,
>> ContentType.create(ent.getContentType())
>> );
>> }
>> return simple;
>> }
>> }
>>
>>
>> Thank you for your time,
>> Scott
>
>Hi Scott
>
>Your code looks correct. However I am wondering why one would want to
>convert streaming messages into simple ones in the first place.
>
>Oleg
Hi Oleg,
There are two reasons I'm interested in turning the streaming responses from the classic client into simple messages:
1) Allowing users to provide very simple in memory parsing solutions which will run against both synchronous and asynchronous modules.
2) Beam has very restrictive serializability requirements. This means not all out of the box streaming consumption tools available can be put to work, at least not without some syntactic tradeoffs, so providing users simple and involved platforms to build off of seems to make the most sense.
Scott
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Converting SimpleHttpRequest to ClassicHttpRequest and
Producing SimpleHttpResponses
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2020-10-06 at 02:20 +0000, Cesar, Scott wrote:
> Hi,
> I'm working to build a streaming HTTP library for Apache Beam. As
> part of providing a low entry effort
> interface we want to provide asynchronous and synchronous options
> which rely on
> SimpleHttpRequest / SimpleHttpResponse.
>
> As the current synchronous client only handles ClassicHttpRequests,
> I'm currently using:
> public static ClassicHttpRequest convert(SimpleHttpRequest original){
> BasicClassicHttpRequest copy = new
> BasicClassicHttpRequest(original.getMethod(),
> original.getRequestUri());
> copy.setVersion(original.getVersion());
> for (final Iterator<Header> it = original.headerIterator();
> it.hasNext(); ) {
> copy.addHeader(it.next());
> }
> copy.setScheme(original.getScheme());
> copy.setAuthority(original.getAuthority());
>
> HttpEntity entity;
> SimpleBody body = original.getBody();
> if(body != null){
> if (body.isBytes()){
> entity = new ByteArrayEntity(body.getBodyBytes(),
> body.getContentType());
> } else{
> entity = new StringEntity(body.getBodyText(),
> body.getContentType());
> }
> copy.setEntity(entity);
> }
> return copy;
> }
>
> To convert simple requests into classic ones. Is this the best way to
> go about it (and if so, would this be
> welcome as a pull request on SimpleHttpRequest), or is there a better
> way to provide this support (e.g.
> extending the classic client to be able to handle SimpleRequests
> directly)
>
> Similarly, I'm using a relatively involved method of greedily
> consuming a ClassicHttpResponse to produce a
> SimpleHttpResponse from the classic client (code included at the
> end), and am curious if there's a better way
> of doing this conversion or, if not, if providing a
> SimpleHttpResponseHandler and making it available from the client
> would be a good way of providing a unified interface going forewards.
>
> public class SimpleHttpResponseHandler implements
> HttpClientResponseHandler<SimpleHttpResponse> {
>
> @Override
> public SimpleHttpResponse handleResponse(ClassicHttpResponse
> response) throws HttpException, IOException {
> SimpleHttpResponse simple =
> SimpleHttpResponse.copy(response);
> HttpEntity ent = response.getEntity();
> if(ent != null){
> byte[] data;
> if (ent.getContentLength() >0) {
> if(ent.getContentLength() > Integer.MAX_VALUE){
> throw new RuntimeException("Received to much data
> to buffer in memory, try using a streaming consumer");
> }
> data = new byte[(int) ent.getContentLength()];
> int read = ent.getContent().read(data);
> if(read != ent.getContentLength())
> throw new RuntimeException("Received "+read+"
> bytes instead of "+ent.getContentLength());
> } else{
> final int bufferSize = 8192;
> InputStream is = ent.getContent();
> ArrayList<byte[]> buffers = new ArrayList<>();
> byte[] swapBuf = new byte[bufferSize];
> int read = is.read(swapBuf);
> while(read != -1 && read == bufferSize){
> buffers.add(swapBuf);
> swapBuf = new byte[bufferSize];
> read = is.read(swapBuf);
> }
>
> int content = bufferSize*buffers.size();
> if(read != -1)
> content += read;
>
> if(content <= 0)
> throw new RuntimeException("Received to many
> bytes to buffer in memory");
> data = new byte[content];
>
> int index = 0;
> for (final byte[] toCpy : buffers){
> System.arraycopy(toCpy, 0, data,
> bufferSize*index, bufferSize);
> index +=1;
> }
> if(read != -1)
> System.arraycopy(swapBuf, 0, data,
> bufferSize*index, bufferSize);
>
> swapBuf = null;
> buffers.clear();
> }
>
> simple.setBody(
> data,
> ContentType.create(ent.getContentType())
> );
> }
> return simple;
> }
> }
>
>
> Thank you for your time,
> Scott
Hi Scott
Your code looks correct. However I am wondering why one would want to
convert streaming messages into simple ones in the first place.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org