[jira] [Created] (SHINDIG-1720) allow proxy handler to handle streamed request which doesn't need shindig server to process the fetched stream

["Marshall Shi (Created) (JIRA)" <ji...@apache.org>]

allow proxy handler to handle streamed request which doesn't need shindig server to process the fetched stream
--------------------------------------------------------------------------------------------------------------

                 Key: SHINDIG-1720
                 URL: https://issues.apache.org/jira/browse/SHINDIG-1720
             Project: Shindig
          Issue Type: New Feature
          Components: Java, Javascript 
    Affects Versions: 2.5.0
            Reporter: Marshall Shi
             Fix For: 2.5.0


In enterprise gadget development, there are requirements to allow files upload/download up to 512MB.  Shindig server is processing the fetched response from target url every time, convert the stream fetched into byte array and store the byte array in HttpResponse, later on the HttpResponse can be copied to a ServletResponse and return to client. This solution has severe problem when the stream is huge: it could easily cause the JVM to exhaust its heap under moderate to heavy load.

Dan's patch about proxied-form-post would solve the upload. https://issues.apache.org/jira/browse/SHINDIG-1695

We'd like to propose a download solution: streamed request. The idea is to add a new field in gadgets.io.RequestParameters.STREAMED, and update gadgets.io.getProxyUrl() to consider the case that this is a streamed request, and return a proxied URL like this: http://localhost:8080/gadgets/proxy?container=default&refresh=0&url=http%3A%2F%2Flocalhost%3A8080%2Ftest%2Ftest.rar&streamed=true

Then Shindig server side ProxyHandler would be able to tell whether the response stream need to be processed in Shindig server or not. If it is a streamed request, Shindig server won't process it, just set the stream into HttpResponse and return it to ServletReponse. All the other processes such as response headers, security, caching etc will be kept the same as they were.

The changes need to be made in Shindig server are Proxy URL handling, HttpResponse, HttpResponseBuilder, ProxyHandler, ProxyServlet, DefaultRequestPipeline and BasicHttpFetcher.

In our project environment, we've injected a our own HttpFetcher, right now, our prototype already work well with those additional changes mentioned above in Shindig. We would like to create a patch for community to review.

The patch will be available for review in 2 week, in the meantime, we would like to get community's comments about this approach.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira