Is pipes really supposed to be a serious API? Is it being actively developed?

[Marc Vaillant <va...@fastmail.fm>]

Only about 5 pipes/c++ related posts since mid July, and basically no
responses.  Is anyone really using or actively developing pipes?  We've
invested some time to make it platform independent (ported bsd sockets
to boost sockets, and the xdr serialization to boost serialization), but
it's still lacking an efficient way to submit jobs, more efficient
mechanism for executing map/reduce without launching pipes executables,
etc.    

Hello, anyone out there?

Marc

Re: Is pipes really supposed to be a serious API? Is it being actively developed?

[Owen O'Malley <ow...@gmail.com>]

I'm sorry that your questions haven't been answered. Pipes is used  
extensively at Yahoo to build the Webmap, which is the graph of the  
entire web. You can now set counters in pipes, but mostly it has just  
been working for Yahoo. It isn't expected to perform better the java  
api, because all of the framework code is the same java. It seems to  
perform very close to the java. If you have patches to make it more  
portable, that would be great.

-- Owen

On Sep 19, 2008, at 11:10 AM, Marc Vaillant <va...@fastmail.fm>  
wrote:

> Only about 5 pipes/c++ related posts since mid July, and basically no
> responses.  Is anyone really using or actively developing pipes?   
> We've
> invested some time to make it platform independent (ported bsd sockets
> to boost sockets, and the xdr serialization to boost serialization),  
> but
> it's still lacking an efficient way to submit jobs, more efficient
> mechanism for executing map/reduce without launching pipes  
> executables,
> etc.
>
> Hello, anyone out there?
>
> Marc
>

Re: Is pipes really supposed to be a serious API? Is it being actively developed?

[David Richards <dr...@showcase60.com>]

I've had the same problem, when wanting to integrate pipes into my  
system.  I haven't seen serious support/comment on pipes, so I'm  
seeing if I can steer clear of this.  Maybe this is a wakeup call to  
see if we've both missed something.

David


On Sep 19, 2008, at 12:10 PM, Marc Vaillant wrote:

> Only about 5 pipes/c++ related posts since mid July, and basically no
> responses.  Is anyone really using or actively developing pipes?   
> We've
> invested some time to make it platform independent (ported bsd sockets
> to boost sockets, and the xdr serialization to boost  
> serialization), but
> it's still lacking an efficient way to submit jobs, more efficient
> mechanism for executing map/reduce without launching pipes  
> executables,
> etc.
>
> Hello, anyone out there?
>
> Marc
>

Re: Is pipes really supposed to be a serious API? Is it being actively developed?

[Alex Feinberg <al...@socialmedia.com>]

Hi Marc,

I have myself personally used pipes, but I found it's performance wasn't faster
than Java (on the same issue). I also found a lack of documentation (e.g. I saw
nothing about xdr).

What I'd like to see is an example where you're using pipes to pass a struct
from a mapper to a reducer using xdr (I have used bit arrays/packed structures
to pass data from a mapper to a reducer using native byte order since it's safe
to assume that machines in a cluster would be heterogenous).

Perhaps if pipes saw more exposure it'd be used more frequently.

Thans,

On Fri, Sep 19, 2008 at 11:10 AM, Marc Vaillant <va...@fastmail.fm> wrote:
> Only about 5 pipes/c++ related posts since mid July, and basically no
> responses.  Is anyone really using or actively developing pipes?  We've
> invested some time to make it platform independent (ported bsd sockets
> to boost sockets, and the xdr serialization to boost serialization), but
> it's still lacking an efficient way to submit jobs, more efficient
> mechanism for executing map/reduce without launching pipes executables,
> etc.
>
> Hello, anyone out there?
>
> Marc
>
>



-- 
Alex Feinberg
Platform Engineer, SocialMedia Networks