Re: transport-level issues

[tomer filiba <to...@gmail.com>]

well i finally had the time to implement thrift over stdpipes. it was all
very straight-forward, except for two pitfalls --
you have to flush after every write(), since pipes are normally
line-buffered, and in the client side,
you have to set stdin to nonblocking (fcntl) and use select() to wait for
input.

by the way, since i need my code to run both on windows and posix, i'll have
to resort to sockets.
it's amazing how lousy the win32 api is. my client will start the server
process, which will open a socket,
and report the port number back to the client over stdout. that's the only
portable way i could find.

so even though i'm dropping it, i'm attaching the code for the benefit of
the community:

### java client ###
    public static class TPipesTransport extends TIOStreamTransport
    {
        public TPipesTransport(InputStream is, OutputStream os)
        {
            super(is, os);
        }

        public void write(byte[] buf, int off, int len)
                throws TTransportException
        {
            super.write(buf, off, len);
            flush();
        }
    }

    public static void main(String[] args)
    {
        try {
            ProcessBuilder pb = new ProcessBuilder("/usr/bin/python", "-u",
"pipeserver.py");
            Process proc = pb.start();

            TPipesTransport transport = new
TPipesTransport(proc.getInputStream(), proc.getOutputStream());
            TProtocol protocol = new TBinaryProtocol(transport);
            MyService.Client client = new MyService.Client(protocol);

            client.get_foo();
            transport.close();
            proc.waitFor();
        } catch (Exception x) {
            x.printStackTrace();
        }

    }

### python server ###

class TPipeTransport(TTransport.TTransportBase):
    def __init__(self, inp, outp):
        self.inp = inp
        self.outp = outp
    def close(self):
        self.inp.close()
        self.outp.close()
    def isOpen(self):
        return not self.inp.closed and not self.outp.closed
    def read(self, sz):
        select([self.inp], [], [])
        data = self.inp.read(sz)
        return data
    def write(self, buf):
        self.outp.write(buf)
        self.outp.flush()
    def flush(self):
        self.outp.flush()

class SingleServer(TServer.TServer):
    def serve(self):
        itrans =
self.inputTransportFactory.getTransport(self.serverTransport)
        otrans =
self.outputTransportFactory.getTransport(self.serverTransport)
        iprot = self.inputProtocolFactory.getProtocol(itrans)
        oprot = self.outputProtocolFactory.getProtocol(otrans)
        while True:
            self.processor.process(iprot, oprot)

if __name__ == "__main__":
    import fcntl, os
    fcntl.fcntl(sys.stdin, fcntl.F_SETFL, os.O_NONBLOCK)

    transport = TPipeTransport(sys.stdin, sys.stdout)
    tfactory = TTransport.TBufferedTransportFactory()
    pfactory = TBinaryProtocol.TBinaryProtocolFactory()
    server = SingleServer(processor, transport, tfactory, pfactory)
    try:
        server.serve()
    except EOFError:
        pass



An NCO and a Gentleman


On Tue, Feb 23, 2010 at 01:20, Mark Slee <ms...@facebook.com> wrote:

> One could certainly implement a TPipeTransport for this quite easily. The
> existing TFDTransport might actually already work fine off the shelf for
> this use case, if you already have file descriptors for the pipes.
>
> There are some tickets open on JIRA regarding adding SSL/TLS support. I
> believe there is work underway on this, though not sure if any of it has hit
> the trunk yet.
>
> http://issues.apache.org/jira/browse/THRIFT-106
> http://issues.apache.org/jira/browse/THRIFT-151
> http://issues.apache.org/jira/browse/THRIFT-181
> http://issues.apache.org/jira/browse/THRIFT-630
>
>
>
> -----Original Message-----
> From: tomer filiba [mailto:tomerfiliba@gmail.com]
> Sent: Thursday, February 18, 2010 6:09 AM
> To: thrift-user@incubator.apache.org
> Subject: transport-level issues
>
> hi
>
> is there support for pipes at the transport level instead of sockets?
> my intention is to fork a child process running another program, and talk
> with the parent process over the standard pipes.
> this way i could get a java program to use my python package.
> i don't want to resort to using sockets, because of performance and
> security
> issues.
>
> and on that note, i've asked this before but no one answered -- how do i
> use
> ssl/tls with thrift?
> assuming i need mutual authentication?
>
>
> -tomer
>
> An NCO and a Gentleman
>