You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Bertrand Delacretaz <bd...@apache.org> on 2016/04/05 15:28:14 UTC
Serving web pages with Beam pipelines?
Hi,
Besides being an incubation mentor for Beam (which is not relevant to
this message) I'm a long-time Apache Sling committer and PMC member -
Sling is a web applications framework [1] where an important task is
generating web pages dynamically, by assembling and transforming atoms
of content. Sling is very modular and "malleable", uses OSGi services
to implement this.
In this context, I'm interested in experimenting with Beam to
implement an HTTP request -> response pipeline (unless people think
this is crazy ;-)
I see how that would work for the HTTP key-value pairs: request
method, path, parameters, headers as input and HTTP status and headers
as output, shouldn't be a problem.
What I don't see is how to handle the binary request/response streams
- are there any Beam examples which would put me on the right track
for accepting a binary stream and returning another one to my HTTP
clients?
-Bertrand
[1] http://sling.apache.org/
Re: Serving web pages with Beam pipelines?
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi JB,
On Tue, Apr 5, 2016 at 5:28 PM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> ...Let me know if I can help. IMHO, Camel is more appropriate....
Ok, I've seen a presentation about this which looked interesting, I'll
have another look!
-Bertrand
Re: Serving web pages with Beam pipelines?
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Tue, Apr 5, 2016 at 5:30 PM, Lukasz Cwik <lc...@google.com> wrote:
> Since Beam is all about being able to distribute work, you'll need an HTTP
> request/response system where the response can be given from any machine.
> This will most likely require some device/service which holds connections
> open for you which you can connect back to serve the response....
Ok, this matches what I was thinking so I'm not too much off-base.
I'll keep the idea in the back of my mind - maybe streams would help
"just" for the task of intelligently routing requests to the right
backend (micro-)services, which matches the Sling processing logic
well. And then use the usual HTTP mechanisms to dispatch requests.
-Bertrand
Re: Serving web pages with Beam pipelines?
Posted by Lukasz Cwik <lc...@google.com>.
Since Beam is all about being able to distribute work, you'll need an HTTP
request/response system where the response can be given from any machine.
This will most likely require some device/service which holds connections
open for you which you can connect back to serve the response. You would
need to have a custom source which "streams" requests from this device and
a custom sink that "streams" back responses. Beam currently is lacking
support for custom unbounded sinks but as always, anything can be done with
ParDo even if it shouldn't.
On Tue, Apr 5, 2016 at 8:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Bertrand,
>
> it sounds like Camel route, as you don't really manipulate the data, and
> you need service/reply.
>
> I'm not sure Beam will match
>
> Anyway, first, we would need an IO exposing a HTTP service, getting the
> PCollections (bounded), doing some Fn, the updated PCollections have to be
> sent back to the HTTP IO, which is not possible AFAIR.
>
> Let me know if I can help. IMHO, Camel is more appropriate.
>
> Regards
> JB
>
>
> On 04/05/2016 03:28 PM, Bertrand Delacretaz wrote:
>
>> Hi,
>>
>> Besides being an incubation mentor for Beam (which is not relevant to
>> this message) I'm a long-time Apache Sling committer and PMC member -
>> Sling is a web applications framework [1] where an important task is
>> generating web pages dynamically, by assembling and transforming atoms
>> of content. Sling is very modular and "malleable", uses OSGi services
>> to implement this.
>>
>> In this context, I'm interested in experimenting with Beam to
>> implement an HTTP request -> response pipeline (unless people think
>> this is crazy ;-)
>>
>> I see how that would work for the HTTP key-value pairs: request
>> method, path, parameters, headers as input and HTTP status and headers
>> as output, shouldn't be a problem.
>>
>> What I don't see is how to handle the binary request/response streams
>> - are there any Beam examples which would put me on the right track
>> for accepting a binary stream and returning another one to my HTTP
>> clients?
>>
>> -Bertrand
>>
>> [1] http://sling.apache.org/
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Serving web pages with Beam pipelines?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Bertrand,
it sounds like Camel route, as you don't really manipulate the data, and
you need service/reply.
I'm not sure Beam will match
Anyway, first, we would need an IO exposing a HTTP service, getting the
PCollections (bounded), doing some Fn, the updated PCollections have to
be sent back to the HTTP IO, which is not possible AFAIR.
Let me know if I can help. IMHO, Camel is more appropriate.
Regards
JB
On 04/05/2016 03:28 PM, Bertrand Delacretaz wrote:
> Hi,
>
> Besides being an incubation mentor for Beam (which is not relevant to
> this message) I'm a long-time Apache Sling committer and PMC member -
> Sling is a web applications framework [1] where an important task is
> generating web pages dynamically, by assembling and transforming atoms
> of content. Sling is very modular and "malleable", uses OSGi services
> to implement this.
>
> In this context, I'm interested in experimenting with Beam to
> implement an HTTP request -> response pipeline (unless people think
> this is crazy ;-)
>
> I see how that would work for the HTTP key-value pairs: request
> method, path, parameters, headers as input and HTTP status and headers
> as output, shouldn't be a problem.
>
> What I don't see is how to handle the binary request/response streams
> - are there any Beam examples which would put me on the right track
> for accepting a binary stream and returning another one to my HTTP
> clients?
>
> -Bertrand
>
> [1] http://sling.apache.org/
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com