You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Michael Hausenblas <mi...@gmail.com> on 2012/09/04 12:16:11 UTC

Thrift?

All,

I'm wondering: Protocol Buffers (and Avro) is mentioned a couple of times both in the proposal and in the git README but no mention of Apache Thrift [1] at all.

Based on uptake and being a top-level Apache project, I think we might want to consider re-using/basing Drill on Thrift, or, at least, mention it as an option?

Cheers,
	   Michael

[1] http://thrift.apache.org/

--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/


Re: Thrift?

Posted by Michael Hausenblas <mi...@gmail.com>.
> I guess it is mainly because Dremel is developed by Google, and explain how to use Protocol Buffer in paper. Thus, Protocol Buffers are adopted in Drill by default. :-)

While I agree with the former I don't agree with the latter (re 'by default') - think Avro ;)


Cheers,
	   Michael

--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/

On 4 Sep 2012, at 16:28, lesliewang wrote:

> 
> Good point. They are quite similar. 
> I guess it is mainly because Dremel is developed by Google, and explain how to use Protocol Buffer in paper. Thus, Protocol Buffers are adopted in Drill by default. :-)
> 
>> From: michael.hausenblas@gmail.com
>> Subject: Thrift?
>> Date: Tue, 4 Sep 2012 11:16:11 +0100
>> To: drill-dev@incubator.apache.org
>> 
>> All,
>> 
>> I'm wondering: Protocol Buffers (and Avro) is mentioned a couple of times both in the proposal and in the git README but no mention of Apache Thrift [1] at all.
>> 
>> Based on uptake and being a top-level Apache project, I think we might want to consider re-using/basing Drill on Thrift, or, at least, mention it as an option?
>> 
>> Cheers,
>> 	   Michael
>> 
>> [1] http://thrift.apache.org/
>> 
>> --
>> Michael Hausenblas
>> Ireland, Europe
>> http://mhausenblas.info/
>> 
> 		 	   		  


Re: Thrift?

Posted by Tomer Shiran <ts...@maprtech.com>.
+1

On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com> wrote:

> Hi folks,
>
> I just commented on this first JIRA.  Here is my text:
>
> This issue has been hashed over a lot in the Hadoop projects. There
> was work done to compare thrift vs avro vs protobuf. The conclusion
> was protobuf was the decision to use.
>
> Prior to this move, there had been a lot of noise about pluggable RPC
> transports, and whatnot. It held up adoption of a backwards compatible
> serialization framework for a long time. The problem ended up being
> the analysis-paralysis, rather than the specific implementation
> problem. In other words, the problem was a LACK of implementation than
> actual REAL problems.
>
> Based on this experience, I'd strongly suggest adopting protobuf and
> moving on. Forget about pluggable RPC implementations, the complexity
> doesnt deliver benefits. The benefits of protobuf is that its the RPC
> format for Hadoop and HBase, which allows Drill to draw on the broad
> experience of those communities who need to implement high performance
> backwards compatible RPC serialization.
>
> ====
>
> Expanding a bit, I've looked in to this issue a lot, and there is very
> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> percent faster of this, and that, etc.  I'd strongly suggest protobuf
> for the expanded community.  There is no particular Apache imperative
> that Apache projects re-use libraries.  Use what makes sense for your
> project.
>
> As regards to Avro, it's a fine serialization format for long term
> data retention, but the complexities that exist to enable that make it
> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
>
> -ryan
>
> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
> wrote:
> > We plan to propose the architecture and interfaces in the next couple
> > weeks, which will make it easy to divide the project into clear building
> > blocks. At that point it will be easier to start contributing different
> > data sources, data formats, operators, query languages, etc.
> >
> > The contributions are done in the usual Apache way. It's best to open a
> > JIRA and then post a patch so that others can review and then a committer
> > can check it in.
> >
> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> chandanmadhesia@gmail.com
> >> wrote:
> >
> >> Hi
> >>
> >> Hi
> >>
> >> What is the process to become a contributor to drill ?
> >>
> >> Regards
> >> chandan
> >>
> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
> wrote:
> >>
> >> > Suffice it to say that if *you* think it is important enough to
> implement
> >> > and maintain, then the group shouldn't say naye.  The consensus stuff
> >> > should only block things that break something else.  Additive features
> >> that
> >> > are highly maintainable (or which come with commitments) shouldn't
> >> > generally be blocked.
> >> >
> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> >> > michael.hausenblas@gmail.com> wrote:
> >> >
> >> > > Good. Feel free to put me down for that, if the group as a whole
> thinks
> >> > > that (supporting Thrift) makes sense.
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> > Tomer Shiran
> > Director of Product Management | MapR Technologies | 650-804-8657
>



-- 
Tomer Shiran
Director of Product Management | MapR Technologies | 650-804-8657

Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
These are for external data.

And people do have external data in these formats.  JSON, in particular, is
probably the simplest nested record format that there is so it would be
nice to support for early testing.

On Sat, Sep 15, 2012 at 2:49 AM, Michael Hausenblas <
michael.hausenblas@gmail.com> wrote:

> The only one thing I really don't get is: why did you put Avro and JSON
> into the proposal [1] in the first place? Or is this the 'external access'
> from above?

Re: Thrift?

Posted by Min Zhou <co...@gmail.com>.
Hi, Ted,

I haven't test on that scenario. IMHO, the efficient gap between java
and c++ is mostly depend on the GC overhead, especially in the server
side.


Thanks,
Min

On Mon, Sep 17, 2012 at 1:19 AM, Ted Dunning <te...@gmail.com> wrote:

> That isn't bad for pure java.
>
> What about calling between c++ and java?
>
> On Sun, Sep 16, 2012 at 9:39 AM, Min Zhou <co...@gmail.com> wrote:
>
> > Hmm.... Actually, I can make the throughput to 171,000 on our
> > implementation.
> > But I can't make it better, because of the cost introduced by gc.
> >
> > Min
> >
> >
> > On Mon, Sep 17, 2012 at 12:34 AM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > > The file server is in C++.  The client can be c++ or Java.  I don't
> know
> > > about speed tests for a Java server.
> > >
> > > On Sun, Sep 16, 2012 at 9:22 AM, Min Zhou <co...@gmail.com> wrote:
> > >
> > > > Really? Is that written in java?
> > > >
> > > > Min
> > > >
> > > > On Mon, Sep 17, 2012 at 12:02 AM, Ted Dunning <ted.dunning@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Similarly, the MapR rpc implementation can do over 250,000 ops per
> > > > second.
> > > > >
> > > > > On Sat, Sep 15, 2012 at 8:59 PM, Min Zhou <co...@gmail.com>
> > wrote:
> > > > >
> > > > > > We've developed another RPC following the best practice how to
> use
> > > java
> > > > > > nio.
> > > > > > Under the power of mina2/netty/grizzly, with a good io
> > thread-model,
> > >  a
> > > > > > good memory management, carefully avoid memory copies, and system
> > > > > > context switches,  we made the throughput up to 168,000 ops.
> > > > > > see http://code.google.com/p/nfs-rpc/
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > My research interests are distributed systems, parallel computing and
> > > > bytecode based virtual machine.
> > > >
> > > > My profile:
> > > > http://www.linkedin.com/in/coderplay
> > > > My blog:
> > > > http://coderplay.javaeye.com
> > > >
> > >
> >
> >
> >
> > --
> > My research interests are distributed systems, parallel computing and
> > bytecode based virtual machine.
> >
> > My profile:
> > http://www.linkedin.com/in/coderplay
> > My blog:
> > http://coderplay.javaeye.com
> >
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
That isn't bad for pure java.

What about calling between c++ and java?

On Sun, Sep 16, 2012 at 9:39 AM, Min Zhou <co...@gmail.com> wrote:

> Hmm.... Actually, I can make the throughput to 171,000 on our
> implementation.
> But I can't make it better, because of the cost introduced by gc.
>
> Min
>
>
> On Mon, Sep 17, 2012 at 12:34 AM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > The file server is in C++.  The client can be c++ or Java.  I don't know
> > about speed tests for a Java server.
> >
> > On Sun, Sep 16, 2012 at 9:22 AM, Min Zhou <co...@gmail.com> wrote:
> >
> > > Really? Is that written in java?
> > >
> > > Min
> > >
> > > On Mon, Sep 17, 2012 at 12:02 AM, Ted Dunning <te...@gmail.com>
> > > wrote:
> > >
> > > > Similarly, the MapR rpc implementation can do over 250,000 ops per
> > > second.
> > > >
> > > > On Sat, Sep 15, 2012 at 8:59 PM, Min Zhou <co...@gmail.com>
> wrote:
> > > >
> > > > > We've developed another RPC following the best practice how to use
> > java
> > > > > nio.
> > > > > Under the power of mina2/netty/grizzly, with a good io
> thread-model,
> >  a
> > > > > good memory management, carefully avoid memory copies, and system
> > > > > context switches,  we made the throughput up to 168,000 ops.
> > > > > see http://code.google.com/p/nfs-rpc/
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > My research interests are distributed systems, parallel computing and
> > > bytecode based virtual machine.
> > >
> > > My profile:
> > > http://www.linkedin.com/in/coderplay
> > > My blog:
> > > http://coderplay.javaeye.com
> > >
> >
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

Re: Thrift?

Posted by Min Zhou <co...@gmail.com>.
Hmm.... Actually, I can make the throughput to 171,000 on our
implementation.
But I can't make it better, because of the cost introduced by gc.

Min


On Mon, Sep 17, 2012 at 12:34 AM, Ted Dunning <te...@gmail.com> wrote:

> The file server is in C++.  The client can be c++ or Java.  I don't know
> about speed tests for a Java server.
>
> On Sun, Sep 16, 2012 at 9:22 AM, Min Zhou <co...@gmail.com> wrote:
>
> > Really? Is that written in java?
> >
> > Min
> >
> > On Mon, Sep 17, 2012 at 12:02 AM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > > Similarly, the MapR rpc implementation can do over 250,000 ops per
> > second.
> > >
> > > On Sat, Sep 15, 2012 at 8:59 PM, Min Zhou <co...@gmail.com> wrote:
> > >
> > > > We've developed another RPC following the best practice how to use
> java
> > > > nio.
> > > > Under the power of mina2/netty/grizzly, with a good io thread-model,
>  a
> > > > good memory management, carefully avoid memory copies, and system
> > > > context switches,  we made the throughput up to 168,000 ops.
> > > > see http://code.google.com/p/nfs-rpc/
> > > >
> > >
> >
> >
> >
> > --
> > My research interests are distributed systems, parallel computing and
> > bytecode based virtual machine.
> >
> > My profile:
> > http://www.linkedin.com/in/coderplay
> > My blog:
> > http://coderplay.javaeye.com
> >
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
The file server is in C++.  The client can be c++ or Java.  I don't know
about speed tests for a Java server.

On Sun, Sep 16, 2012 at 9:22 AM, Min Zhou <co...@gmail.com> wrote:

> Really? Is that written in java?
>
> Min
>
> On Mon, Sep 17, 2012 at 12:02 AM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Similarly, the MapR rpc implementation can do over 250,000 ops per
> second.
> >
> > On Sat, Sep 15, 2012 at 8:59 PM, Min Zhou <co...@gmail.com> wrote:
> >
> > > We've developed another RPC following the best practice how to use java
> > > nio.
> > > Under the power of mina2/netty/grizzly, with a good io thread-model,  a
> > > good memory management, carefully avoid memory copies, and system
> > > context switches,  we made the throughput up to 168,000 ops.
> > > see http://code.google.com/p/nfs-rpc/
> > >
> >
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

Re: Thrift?

Posted by Min Zhou <co...@gmail.com>.
Really? Is that written in java?

Min

On Mon, Sep 17, 2012 at 12:02 AM, Ted Dunning <te...@gmail.com> wrote:

> Similarly, the MapR rpc implementation can do over 250,000 ops per second.
>
> On Sat, Sep 15, 2012 at 8:59 PM, Min Zhou <co...@gmail.com> wrote:
>
> > We've developed another RPC following the best practice how to use java
> > nio.
> > Under the power of mina2/netty/grizzly, with a good io thread-model,  a
> > good memory management, carefully avoid memory copies, and system
> > context switches,  we made the throughput up to 168,000 ops.
> > see http://code.google.com/p/nfs-rpc/
> >
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
Similarly, the MapR rpc implementation can do over 250,000 ops per second.

On Sat, Sep 15, 2012 at 8:59 PM, Min Zhou <co...@gmail.com> wrote:

> We've developed another RPC following the best practice how to use java
> nio.
> Under the power of mina2/netty/grizzly, with a good io thread-model,  a
> good memory management, carefully avoid memory copies, and system
> context switches,  we made the throughput up to 168,000 ops.
> see http://code.google.com/p/nfs-rpc/
>

Re: Thrift?

Posted by Hyunsik Choi <hy...@gmail.com>.
Min,

Thank you for comments.

On Sun, Sep 16, 2012 at 12:59 PM, Min Zhou <co...@gmail.com> wrote:

> Hi, Hyunsik,
>
> Hadoop was born before the java nio became popular. Initially, Hadoop's IPC
> was written in the way of block io. Although, the recent version of hadoop
> and
> Yarn has change its IPC implementation use nio, but for historical reason,
> its
> not a typical way how to use java nio. We did a benchmark on YarnRPC, the
> throughput is no more than 50,000 ops, to be worse, to earn a good result,
> you should increase the number of RPC handlers.
> We've developed another RPC following the best practice how to use java
> nio.
> Under the power of mina2/netty/grizzly, with a good io thread-model,  a
> good memory management, carefully avoid memory copies, and system
> context switches,  we made the throughput up to 168,000 ops.
> see http://code.google.com/p/nfs-rpc/
>
> Thanks,
> Min
>
> On Sun, Sep 16, 2012 at 11:15 AM, Hyunsik Choi <hyunsik.choi@gmail.com
> >wrote:
>
> > Ted,
> >
> > Thank you for detail description. I agree that.
> > In terms of productivity, I mentioned YarnRPC,
> >
> > --
> > Hyunsik Choi
> >
> > On Sun, Sep 16, 2012 at 6:16 AM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > > YarnRPC is based on the original Hadoop RPC.  The major change is the
> > > change from Jute serialization to protobufs.
> > >
> > > Some of the limitations that I know of are:
> > >
> > > - only has a Java implementation
> > >
> > > - uses lots of synchronization instead of using lockless structures
> where
> > > possible
> > >
> > > - it is only client/server, not peer to peer
> > >
> > > - it doesn't support actor-style messages
> > >
> > > I have to admit that I haven't read the details for some time.
> > >
> > > On Sat, Sep 15, 2012 at 6:59 AM, Hyunsik Choi <hyunsik.choi@gmail.com
> > > >wrote:
> > >
> > > > You are right. I missed ProcobufRpcEngine transforms the proto
> message
> > > into
> > > > a RpcResponseWritable object. Currently, It is not portable to other
> > > > languages.
> > > >
> > > > However, why do you think YarnRPC is inefficient? It is just curious
> =)
> > > >
> > > > --
> > > > Hyunsik Choi
> > > >
> > > > On Sat, Sep 15, 2012 at 9:26 PM, Min Zhou <co...@gmail.com>
> wrote:
> > > >
> > > > > YarnRPC -1
> > > > >
> > > > > That's quite inefficient in my experience and doesn't support
> > > > > multi-languages
> > > > > currently.
> > > > >
> > > > > On Sat, Sep 15, 2012 at 7:51 PM, Hyunsik Choi <
> > hyunsik.choi@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > +1 for json as initial data format
> > > > > >
> > > > > > In addition, I recommend YarnRPC with protocol buffer for
> internal
> > > RPC
> > > > > and
> > > > > > API RPC. Protocol buffer is portable to other languages. If we
> use
> > > > > another
> > > > > > RPC system, we have to additionally consider the security aspect
> of
> > > > > Hadoop.
> > > > > >
> > > > > > --
> > > > > > Hyunsik Choi
> > > > > >
> > > > > > On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > There should be 2 types of serialization method. One should
> > define
> > > > its
> > > > > > > schema,
> > > > > > > for the use of RPC,  user wire API; while the other need not
> > define
> > > > > > schema,
> > > > > > > it
> > > > > > > typically for internal data transfer, I think fastjson or kryo
> is
> > > > quite
> > > > > > > suitable for the
> > > > > > > latter purpose.
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Min
> > > > > > >
> > > > > > > On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> > > > > > > michael.hausenblas@gmail.com> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Point taken … +1 for protobuf - from my POV we can close
> > ISSUE-1
> > > > > > > >
> > > > > > > > > The question of an internal wire format, btw, does not
> > > constrain
> > > > > the
> > > > > > > > project relative to external access.
> > > > > > > >
> > > > > > > > Sounds sensible.
> > > > > > > >
> > > > > > > > The only one thing I really don't get is: why did you put
> Avro
> > > and
> > > > > JSON
> > > > > > > > into the proposal [1] in the first place? Or is this the
> > > 'external
> > > > > > > access'
> > > > > > > > from above?
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >            Michael
> > > > > > > >
> > > > > > > > [1] http://wiki.apache.org/incubator/DrillProposal
> > > > > > > >
> > > > > > > > --
> > > > > > > > Michael Hausenblas
> > > > > > > > Ireland, Europe
> > > > > > > > http://mhausenblas.info/
> > > > > > > >
> > > > > > > > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> > > > > > > >
> > > > > > > > > I think that it is important to ask a few questions leading
> > up
> > > a
> > > > > > > decision
> > > > > > > > > here.
> > > > > > > > >
> > > > > > > > > The first is a (rhetorical) show of hands about how many
> > people
> > > > > > believe
> > > > > > > > > that there are no serious performance or expressivity
> killers
> > > > when
> > > > > > > > > comparing alternative serialization frameworks.  As far as
> I
> > > > know,
> > > > > > > > > performance differences are not massive (and protobufs is
> one
> > > of
> > > > > the
> > > > > > > > > leaders in any case) and the expressivity differences are
> > > > > essentially
> > > > > > > > nil.
> > > > > > > > > If somebody feels that there is a serious show-stopper with
> > any
> > > > > > option,
> > > > > > > > > they should speak.
> > > > > > > > >
> > > > > > > > > The second is to ask the sense of the community whether
> they
> > > > judge
> > > > > > > > progress
> > > > > > > > > or perfection in this decision is most important to the
> > > project.
> > > > >  My
> > > > > > > > guess
> > > > > > > > > is that almost everybody would prefer to see progress as
> long
> > > as
> > > > > the
> > > > > > > > > technical choice is not subject to some horrid missing bit.
> > > > > > > > >
> > > > > > > > > The final question is whether it is reasonable to go along
> > with
> > > > > > > protobufs
> > > > > > > > > given that several very experienced engineers prefer it and
> > > would
> > > > > > like
> > > > > > > to
> > > > > > > > > produce code based on it.  If the first two answers are
> > > answered
> > > > to
> > > > > > the
> > > > > > > > > effect of protobufs is about as good as we will find and
> that
> > > > > > progress
> > > > > > > > > trumps small differences, then it seems that moving to
> follow
> > > > this
> > > > > > > > > preference of Jason and Ryan for protobufs might be a
> > > reasonable
> > > > > > thing
> > > > > > > to
> > > > > > > > > do.
> > > > > > > > >
> > > > > > > > > The question of an internal wire format, btw, does not
> > > constrain
> > > > > the
> > > > > > > > > project relative to external access.  I think it is
> important
> > > to
> > > > > > > support
> > > > > > > > > JDBC and ODBC and whatever is in common use for querying.
> >  For
> > > > > > external
> > > > > > > > > access the question is quite different.  Whereas for the
> > > internal
> > > > > > > format
> > > > > > > > > consensus around a single choice has large benefits, the
> > > external
> > > > > > > format
> > > > > > > > > choice is nearly the opposite.  For an external format,
> > > limiting
> > > > > > > > ourselves
> > > > > > > > > to a single choice seems like a bad idea and increasing the
> > > > > audience
> > > > > > > > seems
> > > > > > > > > like a better choice.
> > > > > > > > >
> > > > > > > > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <
> > > > ryanobjc@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi folks,
> > > > > > > > >>
> > > > > > > > >> I just commented on this first JIRA.  Here is my text:
> > > > > > > > >>
> > > > > > > > >> This issue has been hashed over a lot in the Hadoop
> > projects.
> > > > > There
> > > > > > > > >> was work done to compare thrift vs avro vs protobuf. The
> > > > > conclusion
> > > > > > > > >> was protobuf was the decision to use.
> > > > > > > > >>
> > > > > > > > >> Prior to this move, there had been a lot of noise about
> > > > pluggable
> > > > > > RPC
> > > > > > > > >> transports, and whatnot. It held up adoption of a
> backwards
> > > > > > compatible
> > > > > > > > >> serialization framework for a long time. The problem ended
> > up
> > > > > being
> > > > > > > > >> the analysis-paralysis, rather than the specific
> > > implementation
> > > > > > > > >> problem. In other words, the problem was a LACK of
> > > > implementation
> > > > > > than
> > > > > > > > >> actual REAL problems.
> > > > > > > > >>
> > > > > > > > >> Based on this experience, I'd strongly suggest adopting
> > > protobuf
> > > > > and
> > > > > > > > >> moving on. Forget about pluggable RPC implementations, the
> > > > > > complexity
> > > > > > > > >> doesnt deliver benefits. The benefits of protobuf is that
> > its
> > > > the
> > > > > > RPC
> > > > > > > > >> format for Hadoop and HBase, which allows Drill to draw on
> > the
> > > > > broad
> > > > > > > > >> experience of those communities who need to implement high
> > > > > > performance
> > > > > > > > >> backwards compatible RPC serialization.
> > > > > > > > >>
> > > > > > > > >> ====
> > > > > > > > >>
> > > > > > > > >> Expanding a bit, I've looked in to this issue a lot, and
> > there
> > > > is
> > > > > > very
> > > > > > > > >> few significant concrete reasons to choose protobuf vs
> > thrift.
> > > > >  Tiny
> > > > > > > > >> percent faster of this, and that, etc.  I'd strongly
> suggest
> > > > > > protobuf
> > > > > > > > >> for the expanded community.  There is no particular Apache
> > > > > > imperative
> > > > > > > > >> that Apache projects re-use libraries.  Use what makes
> sense
> > > for
> > > > > > your
> > > > > > > > >> project.
> > > > > > > > >>
> > > > > > > > >> As regards to Avro, it's a fine serialization format for
> > long
> > > > term
> > > > > > > > >> data retention, but the complexities that exist to enable
> > that
> > > > > make
> > > > > > it
> > > > > > > > >> non-ideal for an RPC.  I know of no one who uses AvroRPC
> in
> > > any
> > > > > > form.
> > > > > > > > >>
> > > > > > > > >> -ryan
> > > > > > > > >>
> > > > > > > > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> > > > > tshiran@maprtech.com
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>> We plan to propose the architecture and interfaces in the
> > > next
> > > > > > couple
> > > > > > > > >>> weeks, which will make it easy to divide the project into
> > > clear
> > > > > > > > building
> > > > > > > > >>> blocks. At that point it will be easier to start
> > contributing
> > > > > > > different
> > > > > > > > >>> data sources, data formats, operators, query languages,
> > etc.
> > > > > > > > >>>
> > > > > > > > >>> The contributions are done in the usual Apache way. It's
> > best
> > > > to
> > > > > > > open a
> > > > > > > > >>> JIRA and then post a patch so that others can review and
> > > then a
> > > > > > > > committer
> > > > > > > > >>> can check it in.
> > > > > > > > >>>
> > > > > > > > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > > > > > > >> chandanmadhesia@gmail.com
> > > > > > > > >>>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hi
> > > > > > > > >>>>
> > > > > > > > >>>> Hi
> > > > > > > > >>>>
> > > > > > > > >>>> What is the process to become a contributor to drill ?
> > > > > > > > >>>>
> > > > > > > > >>>> Regards
> > > > > > > > >>>> chandan
> > > > > > > > >>>>
> > > > > > > > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> > > > > > ted.dunning@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> Suffice it to say that if *you* think it is important
> > > enough
> > > > to
> > > > > > > > >> implement
> > > > > > > > >>>>> and maintain, then the group shouldn't say naye.  The
> > > > consensus
> > > > > > > stuff
> > > > > > > > >>>>> should only block things that break something else.
> > >  Additive
> > > > > > > > features
> > > > > > > > >>>> that
> > > > > > > > >>>>> are highly maintainable (or which come with
> commitments)
> > > > > > shouldn't
> > > > > > > > >>>>> generally be blocked.
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > > > > > > >>>>> michael.hausenblas@gmail.com> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Good. Feel free to put me down for that, if the group
> > as a
> > > > > whole
> > > > > > > > >> thinks
> > > > > > > > >>>>>> that (supporting Thrift) makes sense.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> --
> > > > > > > > >>> Tomer Shiran
> > > > > > > > >>> Director of Product Management | MapR Technologies |
> > > > 650-804-8657
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > My research interests are distributed systems, parallel
> computing
> > > and
> > > > > > > bytecode based virtual machine.
> > > > > > >
> > > > > > > My profile:
> > > > > > > http://www.linkedin.com/in/coderplay
> > > > > > > My blog:
> > > > > > > http://coderplay.javaeye.com
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > My research interests are distributed systems, parallel computing
> and
> > > > > bytecode based virtual machine.
> > > > >
> > > > > My profile:
> > > > > http://www.linkedin.com/in/coderplay
> > > > > My blog:
> > > > > http://coderplay.javaeye.com
> > > > >
> > > >
> > >
> >
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

Re: Thrift?

Posted by Min Zhou <co...@gmail.com>.
Hi, Hyunsik,

Hadoop was born before the java nio became popular. Initially, Hadoop's IPC
was written in the way of block io. Although, the recent version of hadoop
and
Yarn has change its IPC implementation use nio, but for historical reason,
its
not a typical way how to use java nio. We did a benchmark on YarnRPC, the
throughput is no more than 50,000 ops, to be worse, to earn a good result,
you should increase the number of RPC handlers.
We've developed another RPC following the best practice how to use java nio.
Under the power of mina2/netty/grizzly, with a good io thread-model,  a
good memory management, carefully avoid memory copies, and system
context switches,  we made the throughput up to 168,000 ops.
see http://code.google.com/p/nfs-rpc/

Thanks,
Min

On Sun, Sep 16, 2012 at 11:15 AM, Hyunsik Choi <hy...@gmail.com>wrote:

> Ted,
>
> Thank you for detail description. I agree that.
> In terms of productivity, I mentioned YarnRPC,
>
> --
> Hyunsik Choi
>
> On Sun, Sep 16, 2012 at 6:16 AM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > YarnRPC is based on the original Hadoop RPC.  The major change is the
> > change from Jute serialization to protobufs.
> >
> > Some of the limitations that I know of are:
> >
> > - only has a Java implementation
> >
> > - uses lots of synchronization instead of using lockless structures where
> > possible
> >
> > - it is only client/server, not peer to peer
> >
> > - it doesn't support actor-style messages
> >
> > I have to admit that I haven't read the details for some time.
> >
> > On Sat, Sep 15, 2012 at 6:59 AM, Hyunsik Choi <hyunsik.choi@gmail.com
> > >wrote:
> >
> > > You are right. I missed ProcobufRpcEngine transforms the proto message
> > into
> > > a RpcResponseWritable object. Currently, It is not portable to other
> > > languages.
> > >
> > > However, why do you think YarnRPC is inefficient? It is just curious =)
> > >
> > > --
> > > Hyunsik Choi
> > >
> > > On Sat, Sep 15, 2012 at 9:26 PM, Min Zhou <co...@gmail.com> wrote:
> > >
> > > > YarnRPC -1
> > > >
> > > > That's quite inefficient in my experience and doesn't support
> > > > multi-languages
> > > > currently.
> > > >
> > > > On Sat, Sep 15, 2012 at 7:51 PM, Hyunsik Choi <
> hyunsik.choi@gmail.com
> > > > >wrote:
> > > >
> > > > > +1 for json as initial data format
> > > > >
> > > > > In addition, I recommend YarnRPC with protocol buffer for internal
> > RPC
> > > > and
> > > > > API RPC. Protocol buffer is portable to other languages. If we use
> > > > another
> > > > > RPC system, we have to additionally consider the security aspect of
> > > > Hadoop.
> > > > >
> > > > > --
> > > > > Hyunsik Choi
> > > > >
> > > > > On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com>
> > wrote:
> > > > >
> > > > > > There should be 2 types of serialization method. One should
> define
> > > its
> > > > > > schema,
> > > > > > for the use of RPC,  user wire API; while the other need not
> define
> > > > > schema,
> > > > > > it
> > > > > > typically for internal data transfer, I think fastjson or kryo is
> > > quite
> > > > > > suitable for the
> > > > > > latter purpose.
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Min
> > > > > >
> > > > > > On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> > > > > > michael.hausenblas@gmail.com> wrote:
> > > > > >
> > > > > > >
> > > > > > > Point taken … +1 for protobuf - from my POV we can close
> ISSUE-1
> > > > > > >
> > > > > > > > The question of an internal wire format, btw, does not
> > constrain
> > > > the
> > > > > > > project relative to external access.
> > > > > > >
> > > > > > > Sounds sensible.
> > > > > > >
> > > > > > > The only one thing I really don't get is: why did you put Avro
> > and
> > > > JSON
> > > > > > > into the proposal [1] in the first place? Or is this the
> > 'external
> > > > > > access'
> > > > > > > from above?
> > > > > > >
> > > > > > > Cheers,
> > > > > > >            Michael
> > > > > > >
> > > > > > > [1] http://wiki.apache.org/incubator/DrillProposal
> > > > > > >
> > > > > > > --
> > > > > > > Michael Hausenblas
> > > > > > > Ireland, Europe
> > > > > > > http://mhausenblas.info/
> > > > > > >
> > > > > > > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> > > > > > >
> > > > > > > > I think that it is important to ask a few questions leading
> up
> > a
> > > > > > decision
> > > > > > > > here.
> > > > > > > >
> > > > > > > > The first is a (rhetorical) show of hands about how many
> people
> > > > > believe
> > > > > > > > that there are no serious performance or expressivity killers
> > > when
> > > > > > > > comparing alternative serialization frameworks.  As far as I
> > > know,
> > > > > > > > performance differences are not massive (and protobufs is one
> > of
> > > > the
> > > > > > > > leaders in any case) and the expressivity differences are
> > > > essentially
> > > > > > > nil.
> > > > > > > > If somebody feels that there is a serious show-stopper with
> any
> > > > > option,
> > > > > > > > they should speak.
> > > > > > > >
> > > > > > > > The second is to ask the sense of the community whether they
> > > judge
> > > > > > > progress
> > > > > > > > or perfection in this decision is most important to the
> > project.
> > > >  My
> > > > > > > guess
> > > > > > > > is that almost everybody would prefer to see progress as long
> > as
> > > > the
> > > > > > > > technical choice is not subject to some horrid missing bit.
> > > > > > > >
> > > > > > > > The final question is whether it is reasonable to go along
> with
> > > > > > protobufs
> > > > > > > > given that several very experienced engineers prefer it and
> > would
> > > > > like
> > > > > > to
> > > > > > > > produce code based on it.  If the first two answers are
> > answered
> > > to
> > > > > the
> > > > > > > > effect of protobufs is about as good as we will find and that
> > > > > progress
> > > > > > > > trumps small differences, then it seems that moving to follow
> > > this
> > > > > > > > preference of Jason and Ryan for protobufs might be a
> > reasonable
> > > > > thing
> > > > > > to
> > > > > > > > do.
> > > > > > > >
> > > > > > > > The question of an internal wire format, btw, does not
> > constrain
> > > > the
> > > > > > > > project relative to external access.  I think it is important
> > to
> > > > > > support
> > > > > > > > JDBC and ODBC and whatever is in common use for querying.
>  For
> > > > > external
> > > > > > > > access the question is quite different.  Whereas for the
> > internal
> > > > > > format
> > > > > > > > consensus around a single choice has large benefits, the
> > external
> > > > > > format
> > > > > > > > choice is nearly the opposite.  For an external format,
> > limiting
> > > > > > > ourselves
> > > > > > > > to a single choice seems like a bad idea and increasing the
> > > > audience
> > > > > > > seems
> > > > > > > > like a better choice.
> > > > > > > >
> > > > > > > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <
> > > ryanobjc@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi folks,
> > > > > > > >>
> > > > > > > >> I just commented on this first JIRA.  Here is my text:
> > > > > > > >>
> > > > > > > >> This issue has been hashed over a lot in the Hadoop
> projects.
> > > > There
> > > > > > > >> was work done to compare thrift vs avro vs protobuf. The
> > > > conclusion
> > > > > > > >> was protobuf was the decision to use.
> > > > > > > >>
> > > > > > > >> Prior to this move, there had been a lot of noise about
> > > pluggable
> > > > > RPC
> > > > > > > >> transports, and whatnot. It held up adoption of a backwards
> > > > > compatible
> > > > > > > >> serialization framework for a long time. The problem ended
> up
> > > > being
> > > > > > > >> the analysis-paralysis, rather than the specific
> > implementation
> > > > > > > >> problem. In other words, the problem was a LACK of
> > > implementation
> > > > > than
> > > > > > > >> actual REAL problems.
> > > > > > > >>
> > > > > > > >> Based on this experience, I'd strongly suggest adopting
> > protobuf
> > > > and
> > > > > > > >> moving on. Forget about pluggable RPC implementations, the
> > > > > complexity
> > > > > > > >> doesnt deliver benefits. The benefits of protobuf is that
> its
> > > the
> > > > > RPC
> > > > > > > >> format for Hadoop and HBase, which allows Drill to draw on
> the
> > > > broad
> > > > > > > >> experience of those communities who need to implement high
> > > > > performance
> > > > > > > >> backwards compatible RPC serialization.
> > > > > > > >>
> > > > > > > >> ====
> > > > > > > >>
> > > > > > > >> Expanding a bit, I've looked in to this issue a lot, and
> there
> > > is
> > > > > very
> > > > > > > >> few significant concrete reasons to choose protobuf vs
> thrift.
> > > >  Tiny
> > > > > > > >> percent faster of this, and that, etc.  I'd strongly suggest
> > > > > protobuf
> > > > > > > >> for the expanded community.  There is no particular Apache
> > > > > imperative
> > > > > > > >> that Apache projects re-use libraries.  Use what makes sense
> > for
> > > > > your
> > > > > > > >> project.
> > > > > > > >>
> > > > > > > >> As regards to Avro, it's a fine serialization format for
> long
> > > term
> > > > > > > >> data retention, but the complexities that exist to enable
> that
> > > > make
> > > > > it
> > > > > > > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in
> > any
> > > > > form.
> > > > > > > >>
> > > > > > > >> -ryan
> > > > > > > >>
> > > > > > > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> > > > tshiran@maprtech.com
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >>> We plan to propose the architecture and interfaces in the
> > next
> > > > > couple
> > > > > > > >>> weeks, which will make it easy to divide the project into
> > clear
> > > > > > > building
> > > > > > > >>> blocks. At that point it will be easier to start
> contributing
> > > > > > different
> > > > > > > >>> data sources, data formats, operators, query languages,
> etc.
> > > > > > > >>>
> > > > > > > >>> The contributions are done in the usual Apache way. It's
> best
> > > to
> > > > > > open a
> > > > > > > >>> JIRA and then post a patch so that others can review and
> > then a
> > > > > > > committer
> > > > > > > >>> can check it in.
> > > > > > > >>>
> > > > > > > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > > > > > >> chandanmadhesia@gmail.com
> > > > > > > >>>> wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi
> > > > > > > >>>>
> > > > > > > >>>> Hi
> > > > > > > >>>>
> > > > > > > >>>> What is the process to become a contributor to drill ?
> > > > > > > >>>>
> > > > > > > >>>> Regards
> > > > > > > >>>> chandan
> > > > > > > >>>>
> > > > > > > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> > > > > ted.dunning@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> Suffice it to say that if *you* think it is important
> > enough
> > > to
> > > > > > > >> implement
> > > > > > > >>>>> and maintain, then the group shouldn't say naye.  The
> > > consensus
> > > > > > stuff
> > > > > > > >>>>> should only block things that break something else.
> >  Additive
> > > > > > > features
> > > > > > > >>>> that
> > > > > > > >>>>> are highly maintainable (or which come with commitments)
> > > > > shouldn't
> > > > > > > >>>>> generally be blocked.
> > > > > > > >>>>>
> > > > > > > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > > > > > >>>>> michael.hausenblas@gmail.com> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Good. Feel free to put me down for that, if the group
> as a
> > > > whole
> > > > > > > >> thinks
> > > > > > > >>>>>> that (supporting Thrift) makes sense.
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> Tomer Shiran
> > > > > > > >>> Director of Product Management | MapR Technologies |
> > > 650-804-8657
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > My research interests are distributed systems, parallel computing
> > and
> > > > > > bytecode based virtual machine.
> > > > > >
> > > > > > My profile:
> > > > > > http://www.linkedin.com/in/coderplay
> > > > > > My blog:
> > > > > > http://coderplay.javaeye.com
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > My research interests are distributed systems, parallel computing and
> > > > bytecode based virtual machine.
> > > >
> > > > My profile:
> > > > http://www.linkedin.com/in/coderplay
> > > > My blog:
> > > > http://coderplay.javaeye.com
> > > >
> > >
> >
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Re: Thrift?

Posted by Hyunsik Choi <hy...@gmail.com>.
Ted,

Thank you for detail description. I agree that.
In terms of productivity, I mentioned YarnRPC,

--
Hyunsik Choi

On Sun, Sep 16, 2012 at 6:16 AM, Ted Dunning <te...@gmail.com> wrote:

> YarnRPC is based on the original Hadoop RPC.  The major change is the
> change from Jute serialization to protobufs.
>
> Some of the limitations that I know of are:
>
> - only has a Java implementation
>
> - uses lots of synchronization instead of using lockless structures where
> possible
>
> - it is only client/server, not peer to peer
>
> - it doesn't support actor-style messages
>
> I have to admit that I haven't read the details for some time.
>
> On Sat, Sep 15, 2012 at 6:59 AM, Hyunsik Choi <hyunsik.choi@gmail.com
> >wrote:
>
> > You are right. I missed ProcobufRpcEngine transforms the proto message
> into
> > a RpcResponseWritable object. Currently, It is not portable to other
> > languages.
> >
> > However, why do you think YarnRPC is inefficient? It is just curious =)
> >
> > --
> > Hyunsik Choi
> >
> > On Sat, Sep 15, 2012 at 9:26 PM, Min Zhou <co...@gmail.com> wrote:
> >
> > > YarnRPC -1
> > >
> > > That's quite inefficient in my experience and doesn't support
> > > multi-languages
> > > currently.
> > >
> > > On Sat, Sep 15, 2012 at 7:51 PM, Hyunsik Choi <hyunsik.choi@gmail.com
> > > >wrote:
> > >
> > > > +1 for json as initial data format
> > > >
> > > > In addition, I recommend YarnRPC with protocol buffer for internal
> RPC
> > > and
> > > > API RPC. Protocol buffer is portable to other languages. If we use
> > > another
> > > > RPC system, we have to additionally consider the security aspect of
> > > Hadoop.
> > > >
> > > > --
> > > > Hyunsik Choi
> > > >
> > > > On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com>
> wrote:
> > > >
> > > > > There should be 2 types of serialization method. One should define
> > its
> > > > > schema,
> > > > > for the use of RPC,  user wire API; while the other need not define
> > > > schema,
> > > > > it
> > > > > typically for internal data transfer, I think fastjson or kryo is
> > quite
> > > > > suitable for the
> > > > > latter purpose.
> > > > >
> > > > >
> > > > > Regards,
> > > > > Min
> > > > >
> > > > > On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> > > > > michael.hausenblas@gmail.com> wrote:
> > > > >
> > > > > >
> > > > > > Point taken … +1 for protobuf - from my POV we can close ISSUE-1
> > > > > >
> > > > > > > The question of an internal wire format, btw, does not
> constrain
> > > the
> > > > > > project relative to external access.
> > > > > >
> > > > > > Sounds sensible.
> > > > > >
> > > > > > The only one thing I really don't get is: why did you put Avro
> and
> > > JSON
> > > > > > into the proposal [1] in the first place? Or is this the
> 'external
> > > > > access'
> > > > > > from above?
> > > > > >
> > > > > > Cheers,
> > > > > >            Michael
> > > > > >
> > > > > > [1] http://wiki.apache.org/incubator/DrillProposal
> > > > > >
> > > > > > --
> > > > > > Michael Hausenblas
> > > > > > Ireland, Europe
> > > > > > http://mhausenblas.info/
> > > > > >
> > > > > > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> > > > > >
> > > > > > > I think that it is important to ask a few questions leading up
> a
> > > > > decision
> > > > > > > here.
> > > > > > >
> > > > > > > The first is a (rhetorical) show of hands about how many people
> > > > believe
> > > > > > > that there are no serious performance or expressivity killers
> > when
> > > > > > > comparing alternative serialization frameworks.  As far as I
> > know,
> > > > > > > performance differences are not massive (and protobufs is one
> of
> > > the
> > > > > > > leaders in any case) and the expressivity differences are
> > > essentially
> > > > > > nil.
> > > > > > > If somebody feels that there is a serious show-stopper with any
> > > > option,
> > > > > > > they should speak.
> > > > > > >
> > > > > > > The second is to ask the sense of the community whether they
> > judge
> > > > > > progress
> > > > > > > or perfection in this decision is most important to the
> project.
> > >  My
> > > > > > guess
> > > > > > > is that almost everybody would prefer to see progress as long
> as
> > > the
> > > > > > > technical choice is not subject to some horrid missing bit.
> > > > > > >
> > > > > > > The final question is whether it is reasonable to go along with
> > > > > protobufs
> > > > > > > given that several very experienced engineers prefer it and
> would
> > > > like
> > > > > to
> > > > > > > produce code based on it.  If the first two answers are
> answered
> > to
> > > > the
> > > > > > > effect of protobufs is about as good as we will find and that
> > > > progress
> > > > > > > trumps small differences, then it seems that moving to follow
> > this
> > > > > > > preference of Jason and Ryan for protobufs might be a
> reasonable
> > > > thing
> > > > > to
> > > > > > > do.
> > > > > > >
> > > > > > > The question of an internal wire format, btw, does not
> constrain
> > > the
> > > > > > > project relative to external access.  I think it is important
> to
> > > > > support
> > > > > > > JDBC and ODBC and whatever is in common use for querying.  For
> > > > external
> > > > > > > access the question is quite different.  Whereas for the
> internal
> > > > > format
> > > > > > > consensus around a single choice has large benefits, the
> external
> > > > > format
> > > > > > > choice is nearly the opposite.  For an external format,
> limiting
> > > > > > ourselves
> > > > > > > to a single choice seems like a bad idea and increasing the
> > > audience
> > > > > > seems
> > > > > > > like a better choice.
> > > > > > >
> > > > > > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <
> > ryanobjc@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi folks,
> > > > > > >>
> > > > > > >> I just commented on this first JIRA.  Here is my text:
> > > > > > >>
> > > > > > >> This issue has been hashed over a lot in the Hadoop projects.
> > > There
> > > > > > >> was work done to compare thrift vs avro vs protobuf. The
> > > conclusion
> > > > > > >> was protobuf was the decision to use.
> > > > > > >>
> > > > > > >> Prior to this move, there had been a lot of noise about
> > pluggable
> > > > RPC
> > > > > > >> transports, and whatnot. It held up adoption of a backwards
> > > > compatible
> > > > > > >> serialization framework for a long time. The problem ended up
> > > being
> > > > > > >> the analysis-paralysis, rather than the specific
> implementation
> > > > > > >> problem. In other words, the problem was a LACK of
> > implementation
> > > > than
> > > > > > >> actual REAL problems.
> > > > > > >>
> > > > > > >> Based on this experience, I'd strongly suggest adopting
> protobuf
> > > and
> > > > > > >> moving on. Forget about pluggable RPC implementations, the
> > > > complexity
> > > > > > >> doesnt deliver benefits. The benefits of protobuf is that its
> > the
> > > > RPC
> > > > > > >> format for Hadoop and HBase, which allows Drill to draw on the
> > > broad
> > > > > > >> experience of those communities who need to implement high
> > > > performance
> > > > > > >> backwards compatible RPC serialization.
> > > > > > >>
> > > > > > >> ====
> > > > > > >>
> > > > > > >> Expanding a bit, I've looked in to this issue a lot, and there
> > is
> > > > very
> > > > > > >> few significant concrete reasons to choose protobuf vs thrift.
> > >  Tiny
> > > > > > >> percent faster of this, and that, etc.  I'd strongly suggest
> > > > protobuf
> > > > > > >> for the expanded community.  There is no particular Apache
> > > > imperative
> > > > > > >> that Apache projects re-use libraries.  Use what makes sense
> for
> > > > your
> > > > > > >> project.
> > > > > > >>
> > > > > > >> As regards to Avro, it's a fine serialization format for long
> > term
> > > > > > >> data retention, but the complexities that exist to enable that
> > > make
> > > > it
> > > > > > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in
> any
> > > > form.
> > > > > > >>
> > > > > > >> -ryan
> > > > > > >>
> > > > > > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> > > tshiran@maprtech.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>> We plan to propose the architecture and interfaces in the
> next
> > > > couple
> > > > > > >>> weeks, which will make it easy to divide the project into
> clear
> > > > > > building
> > > > > > >>> blocks. At that point it will be easier to start contributing
> > > > > different
> > > > > > >>> data sources, data formats, operators, query languages, etc.
> > > > > > >>>
> > > > > > >>> The contributions are done in the usual Apache way. It's best
> > to
> > > > > open a
> > > > > > >>> JIRA and then post a patch so that others can review and
> then a
> > > > > > committer
> > > > > > >>> can check it in.
> > > > > > >>>
> > > > > > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > > > > >> chandanmadhesia@gmail.com
> > > > > > >>>> wrote:
> > > > > > >>>
> > > > > > >>>> Hi
> > > > > > >>>>
> > > > > > >>>> Hi
> > > > > > >>>>
> > > > > > >>>> What is the process to become a contributor to drill ?
> > > > > > >>>>
> > > > > > >>>> Regards
> > > > > > >>>> chandan
> > > > > > >>>>
> > > > > > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> > > > ted.dunning@gmail.com>
> > > > > > >> wrote:
> > > > > > >>>>
> > > > > > >>>>> Suffice it to say that if *you* think it is important
> enough
> > to
> > > > > > >> implement
> > > > > > >>>>> and maintain, then the group shouldn't say naye.  The
> > consensus
> > > > > stuff
> > > > > > >>>>> should only block things that break something else.
>  Additive
> > > > > > features
> > > > > > >>>> that
> > > > > > >>>>> are highly maintainable (or which come with commitments)
> > > > shouldn't
> > > > > > >>>>> generally be blocked.
> > > > > > >>>>>
> > > > > > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > > > > >>>>> michael.hausenblas@gmail.com> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Good. Feel free to put me down for that, if the group as a
> > > whole
> > > > > > >> thinks
> > > > > > >>>>>> that (supporting Thrift) makes sense.
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> Tomer Shiran
> > > > > > >>> Director of Product Management | MapR Technologies |
> > 650-804-8657
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > My research interests are distributed systems, parallel computing
> and
> > > > > bytecode based virtual machine.
> > > > >
> > > > > My profile:
> > > > > http://www.linkedin.com/in/coderplay
> > > > > My blog:
> > > > > http://coderplay.javaeye.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > My research interests are distributed systems, parallel computing and
> > > bytecode based virtual machine.
> > >
> > > My profile:
> > > http://www.linkedin.com/in/coderplay
> > > My blog:
> > > http://coderplay.javaeye.com
> > >
> >
>

Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
YarnRPC is based on the original Hadoop RPC.  The major change is the
change from Jute serialization to protobufs.

Some of the limitations that I know of are:

- only has a Java implementation

- uses lots of synchronization instead of using lockless structures where
possible

- it is only client/server, not peer to peer

- it doesn't support actor-style messages

I have to admit that I haven't read the details for some time.

On Sat, Sep 15, 2012 at 6:59 AM, Hyunsik Choi <hy...@gmail.com>wrote:

> You are right. I missed ProcobufRpcEngine transforms the proto message into
> a RpcResponseWritable object. Currently, It is not portable to other
> languages.
>
> However, why do you think YarnRPC is inefficient? It is just curious =)
>
> --
> Hyunsik Choi
>
> On Sat, Sep 15, 2012 at 9:26 PM, Min Zhou <co...@gmail.com> wrote:
>
> > YarnRPC -1
> >
> > That's quite inefficient in my experience and doesn't support
> > multi-languages
> > currently.
> >
> > On Sat, Sep 15, 2012 at 7:51 PM, Hyunsik Choi <hyunsik.choi@gmail.com
> > >wrote:
> >
> > > +1 for json as initial data format
> > >
> > > In addition, I recommend YarnRPC with protocol buffer for internal RPC
> > and
> > > API RPC. Protocol buffer is portable to other languages. If we use
> > another
> > > RPC system, we have to additionally consider the security aspect of
> > Hadoop.
> > >
> > > --
> > > Hyunsik Choi
> > >
> > > On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com> wrote:
> > >
> > > > There should be 2 types of serialization method. One should define
> its
> > > > schema,
> > > > for the use of RPC,  user wire API; while the other need not define
> > > schema,
> > > > it
> > > > typically for internal data transfer, I think fastjson or kryo is
> quite
> > > > suitable for the
> > > > latter purpose.
> > > >
> > > >
> > > > Regards,
> > > > Min
> > > >
> > > > On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> > > > michael.hausenblas@gmail.com> wrote:
> > > >
> > > > >
> > > > > Point taken … +1 for protobuf - from my POV we can close ISSUE-1
> > > > >
> > > > > > The question of an internal wire format, btw, does not constrain
> > the
> > > > > project relative to external access.
> > > > >
> > > > > Sounds sensible.
> > > > >
> > > > > The only one thing I really don't get is: why did you put Avro and
> > JSON
> > > > > into the proposal [1] in the first place? Or is this the 'external
> > > > access'
> > > > > from above?
> > > > >
> > > > > Cheers,
> > > > >            Michael
> > > > >
> > > > > [1] http://wiki.apache.org/incubator/DrillProposal
> > > > >
> > > > > --
> > > > > Michael Hausenblas
> > > > > Ireland, Europe
> > > > > http://mhausenblas.info/
> > > > >
> > > > > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> > > > >
> > > > > > I think that it is important to ask a few questions leading up a
> > > > decision
> > > > > > here.
> > > > > >
> > > > > > The first is a (rhetorical) show of hands about how many people
> > > believe
> > > > > > that there are no serious performance or expressivity killers
> when
> > > > > > comparing alternative serialization frameworks.  As far as I
> know,
> > > > > > performance differences are not massive (and protobufs is one of
> > the
> > > > > > leaders in any case) and the expressivity differences are
> > essentially
> > > > > nil.
> > > > > > If somebody feels that there is a serious show-stopper with any
> > > option,
> > > > > > they should speak.
> > > > > >
> > > > > > The second is to ask the sense of the community whether they
> judge
> > > > > progress
> > > > > > or perfection in this decision is most important to the project.
> >  My
> > > > > guess
> > > > > > is that almost everybody would prefer to see progress as long as
> > the
> > > > > > technical choice is not subject to some horrid missing bit.
> > > > > >
> > > > > > The final question is whether it is reasonable to go along with
> > > > protobufs
> > > > > > given that several very experienced engineers prefer it and would
> > > like
> > > > to
> > > > > > produce code based on it.  If the first two answers are answered
> to
> > > the
> > > > > > effect of protobufs is about as good as we will find and that
> > > progress
> > > > > > trumps small differences, then it seems that moving to follow
> this
> > > > > > preference of Jason and Ryan for protobufs might be a reasonable
> > > thing
> > > > to
> > > > > > do.
> > > > > >
> > > > > > The question of an internal wire format, btw, does not constrain
> > the
> > > > > > project relative to external access.  I think it is important to
> > > > support
> > > > > > JDBC and ODBC and whatever is in common use for querying.  For
> > > external
> > > > > > access the question is quite different.  Whereas for the internal
> > > > format
> > > > > > consensus around a single choice has large benefits, the external
> > > > format
> > > > > > choice is nearly the opposite.  For an external format, limiting
> > > > > ourselves
> > > > > > to a single choice seems like a bad idea and increasing the
> > audience
> > > > > seems
> > > > > > like a better choice.
> > > > > >
> > > > > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <
> ryanobjc@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> I just commented on this first JIRA.  Here is my text:
> > > > > >>
> > > > > >> This issue has been hashed over a lot in the Hadoop projects.
> > There
> > > > > >> was work done to compare thrift vs avro vs protobuf. The
> > conclusion
> > > > > >> was protobuf was the decision to use.
> > > > > >>
> > > > > >> Prior to this move, there had been a lot of noise about
> pluggable
> > > RPC
> > > > > >> transports, and whatnot. It held up adoption of a backwards
> > > compatible
> > > > > >> serialization framework for a long time. The problem ended up
> > being
> > > > > >> the analysis-paralysis, rather than the specific implementation
> > > > > >> problem. In other words, the problem was a LACK of
> implementation
> > > than
> > > > > >> actual REAL problems.
> > > > > >>
> > > > > >> Based on this experience, I'd strongly suggest adopting protobuf
> > and
> > > > > >> moving on. Forget about pluggable RPC implementations, the
> > > complexity
> > > > > >> doesnt deliver benefits. The benefits of protobuf is that its
> the
> > > RPC
> > > > > >> format for Hadoop and HBase, which allows Drill to draw on the
> > broad
> > > > > >> experience of those communities who need to implement high
> > > performance
> > > > > >> backwards compatible RPC serialization.
> > > > > >>
> > > > > >> ====
> > > > > >>
> > > > > >> Expanding a bit, I've looked in to this issue a lot, and there
> is
> > > very
> > > > > >> few significant concrete reasons to choose protobuf vs thrift.
> >  Tiny
> > > > > >> percent faster of this, and that, etc.  I'd strongly suggest
> > > protobuf
> > > > > >> for the expanded community.  There is no particular Apache
> > > imperative
> > > > > >> that Apache projects re-use libraries.  Use what makes sense for
> > > your
> > > > > >> project.
> > > > > >>
> > > > > >> As regards to Avro, it's a fine serialization format for long
> term
> > > > > >> data retention, but the complexities that exist to enable that
> > make
> > > it
> > > > > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any
> > > form.
> > > > > >>
> > > > > >> -ryan
> > > > > >>
> > > > > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> > tshiran@maprtech.com
> > > >
> > > > > >> wrote:
> > > > > >>> We plan to propose the architecture and interfaces in the next
> > > couple
> > > > > >>> weeks, which will make it easy to divide the project into clear
> > > > > building
> > > > > >>> blocks. At that point it will be easier to start contributing
> > > > different
> > > > > >>> data sources, data formats, operators, query languages, etc.
> > > > > >>>
> > > > > >>> The contributions are done in the usual Apache way. It's best
> to
> > > > open a
> > > > > >>> JIRA and then post a patch so that others can review and then a
> > > > > committer
> > > > > >>> can check it in.
> > > > > >>>
> > > > > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > > > >> chandanmadhesia@gmail.com
> > > > > >>>> wrote:
> > > > > >>>
> > > > > >>>> Hi
> > > > > >>>>
> > > > > >>>> Hi
> > > > > >>>>
> > > > > >>>> What is the process to become a contributor to drill ?
> > > > > >>>>
> > > > > >>>> Regards
> > > > > >>>> chandan
> > > > > >>>>
> > > > > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> > > ted.dunning@gmail.com>
> > > > > >> wrote:
> > > > > >>>>
> > > > > >>>>> Suffice it to say that if *you* think it is important enough
> to
> > > > > >> implement
> > > > > >>>>> and maintain, then the group shouldn't say naye.  The
> consensus
> > > > stuff
> > > > > >>>>> should only block things that break something else.  Additive
> > > > > features
> > > > > >>>> that
> > > > > >>>>> are highly maintainable (or which come with commitments)
> > > shouldn't
> > > > > >>>>> generally be blocked.
> > > > > >>>>>
> > > > > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > > > >>>>> michael.hausenblas@gmail.com> wrote:
> > > > > >>>>>
> > > > > >>>>>> Good. Feel free to put me down for that, if the group as a
> > whole
> > > > > >> thinks
> > > > > >>>>>> that (supporting Thrift) makes sense.
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Tomer Shiran
> > > > > >>> Director of Product Management | MapR Technologies |
> 650-804-8657
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > My research interests are distributed systems, parallel computing and
> > > > bytecode based virtual machine.
> > > >
> > > > My profile:
> > > > http://www.linkedin.com/in/coderplay
> > > > My blog:
> > > > http://coderplay.javaeye.com
> > > >
> > >
> >
> >
> >
> > --
> > My research interests are distributed systems, parallel computing and
> > bytecode based virtual machine.
> >
> > My profile:
> > http://www.linkedin.com/in/coderplay
> > My blog:
> > http://coderplay.javaeye.com
> >
>

Re: Thrift?

Posted by Hyunsik Choi <hy...@gmail.com>.
You are right. I missed ProcobufRpcEngine transforms the proto message into
a RpcResponseWritable object. Currently, It is not portable to other
languages.

However, why do you think YarnRPC is inefficient? It is just curious =)

--
Hyunsik Choi

On Sat, Sep 15, 2012 at 9:26 PM, Min Zhou <co...@gmail.com> wrote:

> YarnRPC -1
>
> That's quite inefficient in my experience and doesn't support
> multi-languages
> currently.
>
> On Sat, Sep 15, 2012 at 7:51 PM, Hyunsik Choi <hyunsik.choi@gmail.com
> >wrote:
>
> > +1 for json as initial data format
> >
> > In addition, I recommend YarnRPC with protocol buffer for internal RPC
> and
> > API RPC. Protocol buffer is portable to other languages. If we use
> another
> > RPC system, we have to additionally consider the security aspect of
> Hadoop.
> >
> > --
> > Hyunsik Choi
> >
> > On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com> wrote:
> >
> > > There should be 2 types of serialization method. One should define its
> > > schema,
> > > for the use of RPC,  user wire API; while the other need not define
> > schema,
> > > it
> > > typically for internal data transfer, I think fastjson or kryo is quite
> > > suitable for the
> > > latter purpose.
> > >
> > >
> > > Regards,
> > > Min
> > >
> > > On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> > > michael.hausenblas@gmail.com> wrote:
> > >
> > > >
> > > > Point taken … +1 for protobuf - from my POV we can close ISSUE-1
> > > >
> > > > > The question of an internal wire format, btw, does not constrain
> the
> > > > project relative to external access.
> > > >
> > > > Sounds sensible.
> > > >
> > > > The only one thing I really don't get is: why did you put Avro and
> JSON
> > > > into the proposal [1] in the first place? Or is this the 'external
> > > access'
> > > > from above?
> > > >
> > > > Cheers,
> > > >            Michael
> > > >
> > > > [1] http://wiki.apache.org/incubator/DrillProposal
> > > >
> > > > --
> > > > Michael Hausenblas
> > > > Ireland, Europe
> > > > http://mhausenblas.info/
> > > >
> > > > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> > > >
> > > > > I think that it is important to ask a few questions leading up a
> > > decision
> > > > > here.
> > > > >
> > > > > The first is a (rhetorical) show of hands about how many people
> > believe
> > > > > that there are no serious performance or expressivity killers when
> > > > > comparing alternative serialization frameworks.  As far as I know,
> > > > > performance differences are not massive (and protobufs is one of
> the
> > > > > leaders in any case) and the expressivity differences are
> essentially
> > > > nil.
> > > > > If somebody feels that there is a serious show-stopper with any
> > option,
> > > > > they should speak.
> > > > >
> > > > > The second is to ask the sense of the community whether they judge
> > > > progress
> > > > > or perfection in this decision is most important to the project.
>  My
> > > > guess
> > > > > is that almost everybody would prefer to see progress as long as
> the
> > > > > technical choice is not subject to some horrid missing bit.
> > > > >
> > > > > The final question is whether it is reasonable to go along with
> > > protobufs
> > > > > given that several very experienced engineers prefer it and would
> > like
> > > to
> > > > > produce code based on it.  If the first two answers are answered to
> > the
> > > > > effect of protobufs is about as good as we will find and that
> > progress
> > > > > trumps small differences, then it seems that moving to follow this
> > > > > preference of Jason and Ryan for protobufs might be a reasonable
> > thing
> > > to
> > > > > do.
> > > > >
> > > > > The question of an internal wire format, btw, does not constrain
> the
> > > > > project relative to external access.  I think it is important to
> > > support
> > > > > JDBC and ODBC and whatever is in common use for querying.  For
> > external
> > > > > access the question is quite different.  Whereas for the internal
> > > format
> > > > > consensus around a single choice has large benefits, the external
> > > format
> > > > > choice is nearly the opposite.  For an external format, limiting
> > > > ourselves
> > > > > to a single choice seems like a bad idea and increasing the
> audience
> > > > seems
> > > > > like a better choice.
> > > > >
> > > > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> > > > wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> I just commented on this first JIRA.  Here is my text:
> > > > >>
> > > > >> This issue has been hashed over a lot in the Hadoop projects.
> There
> > > > >> was work done to compare thrift vs avro vs protobuf. The
> conclusion
> > > > >> was protobuf was the decision to use.
> > > > >>
> > > > >> Prior to this move, there had been a lot of noise about pluggable
> > RPC
> > > > >> transports, and whatnot. It held up adoption of a backwards
> > compatible
> > > > >> serialization framework for a long time. The problem ended up
> being
> > > > >> the analysis-paralysis, rather than the specific implementation
> > > > >> problem. In other words, the problem was a LACK of implementation
> > than
> > > > >> actual REAL problems.
> > > > >>
> > > > >> Based on this experience, I'd strongly suggest adopting protobuf
> and
> > > > >> moving on. Forget about pluggable RPC implementations, the
> > complexity
> > > > >> doesnt deliver benefits. The benefits of protobuf is that its the
> > RPC
> > > > >> format for Hadoop and HBase, which allows Drill to draw on the
> broad
> > > > >> experience of those communities who need to implement high
> > performance
> > > > >> backwards compatible RPC serialization.
> > > > >>
> > > > >> ====
> > > > >>
> > > > >> Expanding a bit, I've looked in to this issue a lot, and there is
> > very
> > > > >> few significant concrete reasons to choose protobuf vs thrift.
>  Tiny
> > > > >> percent faster of this, and that, etc.  I'd strongly suggest
> > protobuf
> > > > >> for the expanded community.  There is no particular Apache
> > imperative
> > > > >> that Apache projects re-use libraries.  Use what makes sense for
> > your
> > > > >> project.
> > > > >>
> > > > >> As regards to Avro, it's a fine serialization format for long term
> > > > >> data retention, but the complexities that exist to enable that
> make
> > it
> > > > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any
> > form.
> > > > >>
> > > > >> -ryan
> > > > >>
> > > > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> tshiran@maprtech.com
> > >
> > > > >> wrote:
> > > > >>> We plan to propose the architecture and interfaces in the next
> > couple
> > > > >>> weeks, which will make it easy to divide the project into clear
> > > > building
> > > > >>> blocks. At that point it will be easier to start contributing
> > > different
> > > > >>> data sources, data formats, operators, query languages, etc.
> > > > >>>
> > > > >>> The contributions are done in the usual Apache way. It's best to
> > > open a
> > > > >>> JIRA and then post a patch so that others can review and then a
> > > > committer
> > > > >>> can check it in.
> > > > >>>
> > > > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > > >> chandanmadhesia@gmail.com
> > > > >>>> wrote:
> > > > >>>
> > > > >>>> Hi
> > > > >>>>
> > > > >>>> Hi
> > > > >>>>
> > > > >>>> What is the process to become a contributor to drill ?
> > > > >>>>
> > > > >>>> Regards
> > > > >>>> chandan
> > > > >>>>
> > > > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> > ted.dunning@gmail.com>
> > > > >> wrote:
> > > > >>>>
> > > > >>>>> Suffice it to say that if *you* think it is important enough to
> > > > >> implement
> > > > >>>>> and maintain, then the group shouldn't say naye.  The consensus
> > > stuff
> > > > >>>>> should only block things that break something else.  Additive
> > > > features
> > > > >>>> that
> > > > >>>>> are highly maintainable (or which come with commitments)
> > shouldn't
> > > > >>>>> generally be blocked.
> > > > >>>>>
> > > > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > > >>>>> michael.hausenblas@gmail.com> wrote:
> > > > >>>>>
> > > > >>>>>> Good. Feel free to put me down for that, if the group as a
> whole
> > > > >> thinks
> > > > >>>>>> that (supporting Thrift) makes sense.
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Tomer Shiran
> > > > >>> Director of Product Management | MapR Technologies | 650-804-8657
> > > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > My research interests are distributed systems, parallel computing and
> > > bytecode based virtual machine.
> > >
> > > My profile:
> > > http://www.linkedin.com/in/coderplay
> > > My blog:
> > > http://coderplay.javaeye.com
> > >
> >
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

Re: Thrift?

Posted by Prasanth J <bu...@gmail.com>.
Hello everyone

I saw this nice video long back and would like to share with everyone. eBay presented comparison
of various serialization techniques, comparing their performance for different payloads, serialized
size etc.. 

Presentation:
http://qconsf.com/dl/qcon-sanfran-2011/slides/SastryMalladi_DealingWithPerformanceChallengesOptimizedSerializationTechniques.pdf
Video:
http://www.infoq.com/presentations/Dealing-with-Performance-Challenges-Optimized-Data-Formats

Protobuf performs well in all criteria esp. under high payload size with better size reduction which are critical for drill. 

Thanks
-- Prasanth

On Sep 15, 2012, at 9:42 AM, Ted Dunning <te...@gmail.com> wrote:

> Yarn is also strictly client/server which leads to all kinds of problems in
> Hadoop.
> 
> YarnRPC -1
> 
> On Sat, Sep 15, 2012 at 5:26 AM, Min Zhou <co...@gmail.com> wrote:
> 
>> YarnRPC -1
>> 
>> That's quite inefficient in my experience and doesn't support
>> multi-languages
>> currently.
>> 
>> On Sat, Sep 15, 2012 at 7:51 PM, Hyunsik Choi <hyunsik.choi@gmail.com
>>> wrote:
>> 
>>> +1 for json as initial data format
>>> 
>>> In addition, I recommend YarnRPC with protocol buffer for internal RPC
>> and
>>> API RPC. Protocol buffer is portable to other languages. If we use
>> another
>>> RPC system, we have to additionally consider the security aspect of
>> Hadoop.
>>> 
>>> --
>>> Hyunsik Choi
>>> 
>>> On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com> wrote:
>>> 
>>>> There should be 2 types of serialization method. One should define its
>>>> schema,
>>>> for the use of RPC,  user wire API; while the other need not define
>>> schema,
>>>> it
>>>> typically for internal data transfer, I think fastjson or kryo is quite
>>>> suitable for the
>>>> latter purpose.
>>>> 
>>>> 
>>>> Regards,
>>>> Min
>>>> 
>>>> On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
>>>> michael.hausenblas@gmail.com> wrote:
>>>> 
>>>>> 
>>>>> Point taken … +1 for protobuf - from my POV we can close ISSUE-1
>>>>> 
>>>>>> The question of an internal wire format, btw, does not constrain
>> the
>>>>> project relative to external access.
>>>>> 
>>>>> Sounds sensible.
>>>>> 
>>>>> The only one thing I really don't get is: why did you put Avro and
>> JSON
>>>>> into the proposal [1] in the first place? Or is this the 'external
>>>> access'
>>>>> from above?
>>>>> 
>>>>> Cheers,
>>>>>           Michael
>>>>> 
>>>>> [1] http://wiki.apache.org/incubator/DrillProposal
>>>>> 
>>>>> --
>>>>> Michael Hausenblas
>>>>> Ireland, Europe
>>>>> http://mhausenblas.info/
>>>>> 
>>>>> On 14 Sep 2012, at 22:31, Ted Dunning wrote:
>>>>> 
>>>>>> I think that it is important to ask a few questions leading up a
>>>> decision
>>>>>> here.
>>>>>> 
>>>>>> The first is a (rhetorical) show of hands about how many people
>>> believe
>>>>>> that there are no serious performance or expressivity killers when
>>>>>> comparing alternative serialization frameworks.  As far as I know,
>>>>>> performance differences are not massive (and protobufs is one of
>> the
>>>>>> leaders in any case) and the expressivity differences are
>> essentially
>>>>> nil.
>>>>>> If somebody feels that there is a serious show-stopper with any
>>> option,
>>>>>> they should speak.
>>>>>> 
>>>>>> The second is to ask the sense of the community whether they judge
>>>>> progress
>>>>>> or perfection in this decision is most important to the project.
>> My
>>>>> guess
>>>>>> is that almost everybody would prefer to see progress as long as
>> the
>>>>>> technical choice is not subject to some horrid missing bit.
>>>>>> 
>>>>>> The final question is whether it is reasonable to go along with
>>>> protobufs
>>>>>> given that several very experienced engineers prefer it and would
>>> like
>>>> to
>>>>>> produce code based on it.  If the first two answers are answered to
>>> the
>>>>>> effect of protobufs is about as good as we will find and that
>>> progress
>>>>>> trumps small differences, then it seems that moving to follow this
>>>>>> preference of Jason and Ryan for protobufs might be a reasonable
>>> thing
>>>> to
>>>>>> do.
>>>>>> 
>>>>>> The question of an internal wire format, btw, does not constrain
>> the
>>>>>> project relative to external access.  I think it is important to
>>>> support
>>>>>> JDBC and ODBC and whatever is in common use for querying.  For
>>> external
>>>>>> access the question is quite different.  Whereas for the internal
>>>> format
>>>>>> consensus around a single choice has large benefits, the external
>>>> format
>>>>>> choice is nearly the opposite.  For an external format, limiting
>>>>> ourselves
>>>>>> to a single choice seems like a bad idea and increasing the
>> audience
>>>>> seems
>>>>>> like a better choice.
>>>>>> 
>>>>>> On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> Hi folks,
>>>>>>> 
>>>>>>> I just commented on this first JIRA.  Here is my text:
>>>>>>> 
>>>>>>> This issue has been hashed over a lot in the Hadoop projects.
>> There
>>>>>>> was work done to compare thrift vs avro vs protobuf. The
>> conclusion
>>>>>>> was protobuf was the decision to use.
>>>>>>> 
>>>>>>> Prior to this move, there had been a lot of noise about pluggable
>>> RPC
>>>>>>> transports, and whatnot. It held up adoption of a backwards
>>> compatible
>>>>>>> serialization framework for a long time. The problem ended up
>> being
>>>>>>> the analysis-paralysis, rather than the specific implementation
>>>>>>> problem. In other words, the problem was a LACK of implementation
>>> than
>>>>>>> actual REAL problems.
>>>>>>> 
>>>>>>> Based on this experience, I'd strongly suggest adopting protobuf
>> and
>>>>>>> moving on. Forget about pluggable RPC implementations, the
>>> complexity
>>>>>>> doesnt deliver benefits. The benefits of protobuf is that its the
>>> RPC
>>>>>>> format for Hadoop and HBase, which allows Drill to draw on the
>> broad
>>>>>>> experience of those communities who need to implement high
>>> performance
>>>>>>> backwards compatible RPC serialization.
>>>>>>> 
>>>>>>> ====
>>>>>>> 
>>>>>>> Expanding a bit, I've looked in to this issue a lot, and there is
>>> very
>>>>>>> few significant concrete reasons to choose protobuf vs thrift.
>> Tiny
>>>>>>> percent faster of this, and that, etc.  I'd strongly suggest
>>> protobuf
>>>>>>> for the expanded community.  There is no particular Apache
>>> imperative
>>>>>>> that Apache projects re-use libraries.  Use what makes sense for
>>> your
>>>>>>> project.
>>>>>>> 
>>>>>>> As regards to Avro, it's a fine serialization format for long term
>>>>>>> data retention, but the complexities that exist to enable that
>> make
>>> it
>>>>>>> non-ideal for an RPC.  I know of no one who uses AvroRPC in any
>>> form.
>>>>>>> 
>>>>>>> -ryan
>>>>>>> 
>>>>>>> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
>> tshiran@maprtech.com
>>>> 
>>>>>>> wrote:
>>>>>>>> We plan to propose the architecture and interfaces in the next
>>> couple
>>>>>>>> weeks, which will make it easy to divide the project into clear
>>>>> building
>>>>>>>> blocks. At that point it will be easier to start contributing
>>>> different
>>>>>>>> data sources, data formats, operators, query languages, etc.
>>>>>>>> 
>>>>>>>> The contributions are done in the usual Apache way. It's best to
>>>> open a
>>>>>>>> JIRA and then post a patch so that others can review and then a
>>>>> committer
>>>>>>>> can check it in.
>>>>>>>> 
>>>>>>>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
>>>>>>> chandanmadhesia@gmail.com
>>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi
>>>>>>>>> 
>>>>>>>>> Hi
>>>>>>>>> 
>>>>>>>>> What is the process to become a contributor to drill ?
>>>>>>>>> 
>>>>>>>>> Regards
>>>>>>>>> chandan
>>>>>>>>> 
>>>>>>>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
>>> ted.dunning@gmail.com>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Suffice it to say that if *you* think it is important enough to
>>>>>>> implement
>>>>>>>>>> and maintain, then the group shouldn't say naye.  The consensus
>>>> stuff
>>>>>>>>>> should only block things that break something else.  Additive
>>>>> features
>>>>>>>>> that
>>>>>>>>>> are highly maintainable (or which come with commitments)
>>> shouldn't
>>>>>>>>>> generally be blocked.
>>>>>>>>>> 
>>>>>>>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
>>>>>>>>>> michael.hausenblas@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Good. Feel free to put me down for that, if the group as a
>> whole
>>>>>>> thinks
>>>>>>>>>>> that (supporting Thrift) makes sense.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Tomer Shiran
>>>>>>>> Director of Product Management | MapR Technologies |
>> 650-804-8657
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> My research interests are distributed systems, parallel computing and
>>>> bytecode based virtual machine.
>>>> 
>>>> My profile:
>>>> http://www.linkedin.com/in/coderplay
>>>> My blog:
>>>> http://coderplay.javaeye.com
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode based virtual machine.
>> 
>> My profile:
>> http://www.linkedin.com/in/coderplay
>> My blog:
>> http://coderplay.javaeye.com
>> 


Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
Yarn is also strictly client/server which leads to all kinds of problems in
Hadoop.

YarnRPC -1

On Sat, Sep 15, 2012 at 5:26 AM, Min Zhou <co...@gmail.com> wrote:

> YarnRPC -1
>
> That's quite inefficient in my experience and doesn't support
> multi-languages
> currently.
>
> On Sat, Sep 15, 2012 at 7:51 PM, Hyunsik Choi <hyunsik.choi@gmail.com
> >wrote:
>
> > +1 for json as initial data format
> >
> > In addition, I recommend YarnRPC with protocol buffer for internal RPC
> and
> > API RPC. Protocol buffer is portable to other languages. If we use
> another
> > RPC system, we have to additionally consider the security aspect of
> Hadoop.
> >
> > --
> > Hyunsik Choi
> >
> > On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com> wrote:
> >
> > > There should be 2 types of serialization method. One should define its
> > > schema,
> > > for the use of RPC,  user wire API; while the other need not define
> > schema,
> > > it
> > > typically for internal data transfer, I think fastjson or kryo is quite
> > > suitable for the
> > > latter purpose.
> > >
> > >
> > > Regards,
> > > Min
> > >
> > > On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> > > michael.hausenblas@gmail.com> wrote:
> > >
> > > >
> > > > Point taken … +1 for protobuf - from my POV we can close ISSUE-1
> > > >
> > > > > The question of an internal wire format, btw, does not constrain
> the
> > > > project relative to external access.
> > > >
> > > > Sounds sensible.
> > > >
> > > > The only one thing I really don't get is: why did you put Avro and
> JSON
> > > > into the proposal [1] in the first place? Or is this the 'external
> > > access'
> > > > from above?
> > > >
> > > > Cheers,
> > > >            Michael
> > > >
> > > > [1] http://wiki.apache.org/incubator/DrillProposal
> > > >
> > > > --
> > > > Michael Hausenblas
> > > > Ireland, Europe
> > > > http://mhausenblas.info/
> > > >
> > > > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> > > >
> > > > > I think that it is important to ask a few questions leading up a
> > > decision
> > > > > here.
> > > > >
> > > > > The first is a (rhetorical) show of hands about how many people
> > believe
> > > > > that there are no serious performance or expressivity killers when
> > > > > comparing alternative serialization frameworks.  As far as I know,
> > > > > performance differences are not massive (and protobufs is one of
> the
> > > > > leaders in any case) and the expressivity differences are
> essentially
> > > > nil.
> > > > > If somebody feels that there is a serious show-stopper with any
> > option,
> > > > > they should speak.
> > > > >
> > > > > The second is to ask the sense of the community whether they judge
> > > > progress
> > > > > or perfection in this decision is most important to the project.
>  My
> > > > guess
> > > > > is that almost everybody would prefer to see progress as long as
> the
> > > > > technical choice is not subject to some horrid missing bit.
> > > > >
> > > > > The final question is whether it is reasonable to go along with
> > > protobufs
> > > > > given that several very experienced engineers prefer it and would
> > like
> > > to
> > > > > produce code based on it.  If the first two answers are answered to
> > the
> > > > > effect of protobufs is about as good as we will find and that
> > progress
> > > > > trumps small differences, then it seems that moving to follow this
> > > > > preference of Jason and Ryan for protobufs might be a reasonable
> > thing
> > > to
> > > > > do.
> > > > >
> > > > > The question of an internal wire format, btw, does not constrain
> the
> > > > > project relative to external access.  I think it is important to
> > > support
> > > > > JDBC and ODBC and whatever is in common use for querying.  For
> > external
> > > > > access the question is quite different.  Whereas for the internal
> > > format
> > > > > consensus around a single choice has large benefits, the external
> > > format
> > > > > choice is nearly the opposite.  For an external format, limiting
> > > > ourselves
> > > > > to a single choice seems like a bad idea and increasing the
> audience
> > > > seems
> > > > > like a better choice.
> > > > >
> > > > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> > > > wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> I just commented on this first JIRA.  Here is my text:
> > > > >>
> > > > >> This issue has been hashed over a lot in the Hadoop projects.
> There
> > > > >> was work done to compare thrift vs avro vs protobuf. The
> conclusion
> > > > >> was protobuf was the decision to use.
> > > > >>
> > > > >> Prior to this move, there had been a lot of noise about pluggable
> > RPC
> > > > >> transports, and whatnot. It held up adoption of a backwards
> > compatible
> > > > >> serialization framework for a long time. The problem ended up
> being
> > > > >> the analysis-paralysis, rather than the specific implementation
> > > > >> problem. In other words, the problem was a LACK of implementation
> > than
> > > > >> actual REAL problems.
> > > > >>
> > > > >> Based on this experience, I'd strongly suggest adopting protobuf
> and
> > > > >> moving on. Forget about pluggable RPC implementations, the
> > complexity
> > > > >> doesnt deliver benefits. The benefits of protobuf is that its the
> > RPC
> > > > >> format for Hadoop and HBase, which allows Drill to draw on the
> broad
> > > > >> experience of those communities who need to implement high
> > performance
> > > > >> backwards compatible RPC serialization.
> > > > >>
> > > > >> ====
> > > > >>
> > > > >> Expanding a bit, I've looked in to this issue a lot, and there is
> > very
> > > > >> few significant concrete reasons to choose protobuf vs thrift.
>  Tiny
> > > > >> percent faster of this, and that, etc.  I'd strongly suggest
> > protobuf
> > > > >> for the expanded community.  There is no particular Apache
> > imperative
> > > > >> that Apache projects re-use libraries.  Use what makes sense for
> > your
> > > > >> project.
> > > > >>
> > > > >> As regards to Avro, it's a fine serialization format for long term
> > > > >> data retention, but the complexities that exist to enable that
> make
> > it
> > > > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any
> > form.
> > > > >>
> > > > >> -ryan
> > > > >>
> > > > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> tshiran@maprtech.com
> > >
> > > > >> wrote:
> > > > >>> We plan to propose the architecture and interfaces in the next
> > couple
> > > > >>> weeks, which will make it easy to divide the project into clear
> > > > building
> > > > >>> blocks. At that point it will be easier to start contributing
> > > different
> > > > >>> data sources, data formats, operators, query languages, etc.
> > > > >>>
> > > > >>> The contributions are done in the usual Apache way. It's best to
> > > open a
> > > > >>> JIRA and then post a patch so that others can review and then a
> > > > committer
> > > > >>> can check it in.
> > > > >>>
> > > > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > > >> chandanmadhesia@gmail.com
> > > > >>>> wrote:
> > > > >>>
> > > > >>>> Hi
> > > > >>>>
> > > > >>>> Hi
> > > > >>>>
> > > > >>>> What is the process to become a contributor to drill ?
> > > > >>>>
> > > > >>>> Regards
> > > > >>>> chandan
> > > > >>>>
> > > > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> > ted.dunning@gmail.com>
> > > > >> wrote:
> > > > >>>>
> > > > >>>>> Suffice it to say that if *you* think it is important enough to
> > > > >> implement
> > > > >>>>> and maintain, then the group shouldn't say naye.  The consensus
> > > stuff
> > > > >>>>> should only block things that break something else.  Additive
> > > > features
> > > > >>>> that
> > > > >>>>> are highly maintainable (or which come with commitments)
> > shouldn't
> > > > >>>>> generally be blocked.
> > > > >>>>>
> > > > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > > >>>>> michael.hausenblas@gmail.com> wrote:
> > > > >>>>>
> > > > >>>>>> Good. Feel free to put me down for that, if the group as a
> whole
> > > > >> thinks
> > > > >>>>>> that (supporting Thrift) makes sense.
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Tomer Shiran
> > > > >>> Director of Product Management | MapR Technologies |
> 650-804-8657
> > > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > My research interests are distributed systems, parallel computing and
> > > bytecode based virtual machine.
> > >
> > > My profile:
> > > http://www.linkedin.com/in/coderplay
> > > My blog:
> > > http://coderplay.javaeye.com
> > >
> >
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

Re: Thrift?

Posted by Min Zhou <co...@gmail.com>.
YarnRPC -1

That's quite inefficient in my experience and doesn't support
multi-languages
currently.

On Sat, Sep 15, 2012 at 7:51 PM, Hyunsik Choi <hy...@gmail.com>wrote:

> +1 for json as initial data format
>
> In addition, I recommend YarnRPC with protocol buffer for internal RPC and
> API RPC. Protocol buffer is portable to other languages. If we use another
> RPC system, we have to additionally consider the security aspect of Hadoop.
>
> --
> Hyunsik Choi
>
> On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com> wrote:
>
> > There should be 2 types of serialization method. One should define its
> > schema,
> > for the use of RPC,  user wire API; while the other need not define
> schema,
> > it
> > typically for internal data transfer, I think fastjson or kryo is quite
> > suitable for the
> > latter purpose.
> >
> >
> > Regards,
> > Min
> >
> > On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> > michael.hausenblas@gmail.com> wrote:
> >
> > >
> > > Point taken … +1 for protobuf - from my POV we can close ISSUE-1
> > >
> > > > The question of an internal wire format, btw, does not constrain the
> > > project relative to external access.
> > >
> > > Sounds sensible.
> > >
> > > The only one thing I really don't get is: why did you put Avro and JSON
> > > into the proposal [1] in the first place? Or is this the 'external
> > access'
> > > from above?
> > >
> > > Cheers,
> > >            Michael
> > >
> > > [1] http://wiki.apache.org/incubator/DrillProposal
> > >
> > > --
> > > Michael Hausenblas
> > > Ireland, Europe
> > > http://mhausenblas.info/
> > >
> > > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> > >
> > > > I think that it is important to ask a few questions leading up a
> > decision
> > > > here.
> > > >
> > > > The first is a (rhetorical) show of hands about how many people
> believe
> > > > that there are no serious performance or expressivity killers when
> > > > comparing alternative serialization frameworks.  As far as I know,
> > > > performance differences are not massive (and protobufs is one of the
> > > > leaders in any case) and the expressivity differences are essentially
> > > nil.
> > > > If somebody feels that there is a serious show-stopper with any
> option,
> > > > they should speak.
> > > >
> > > > The second is to ask the sense of the community whether they judge
> > > progress
> > > > or perfection in this decision is most important to the project.  My
> > > guess
> > > > is that almost everybody would prefer to see progress as long as the
> > > > technical choice is not subject to some horrid missing bit.
> > > >
> > > > The final question is whether it is reasonable to go along with
> > protobufs
> > > > given that several very experienced engineers prefer it and would
> like
> > to
> > > > produce code based on it.  If the first two answers are answered to
> the
> > > > effect of protobufs is about as good as we will find and that
> progress
> > > > trumps small differences, then it seems that moving to follow this
> > > > preference of Jason and Ryan for protobufs might be a reasonable
> thing
> > to
> > > > do.
> > > >
> > > > The question of an internal wire format, btw, does not constrain the
> > > > project relative to external access.  I think it is important to
> > support
> > > > JDBC and ODBC and whatever is in common use for querying.  For
> external
> > > > access the question is quite different.  Whereas for the internal
> > format
> > > > consensus around a single choice has large benefits, the external
> > format
> > > > choice is nearly the opposite.  For an external format, limiting
> > > ourselves
> > > > to a single choice seems like a bad idea and increasing the audience
> > > seems
> > > > like a better choice.
> > > >
> > > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> I just commented on this first JIRA.  Here is my text:
> > > >>
> > > >> This issue has been hashed over a lot in the Hadoop projects. There
> > > >> was work done to compare thrift vs avro vs protobuf. The conclusion
> > > >> was protobuf was the decision to use.
> > > >>
> > > >> Prior to this move, there had been a lot of noise about pluggable
> RPC
> > > >> transports, and whatnot. It held up adoption of a backwards
> compatible
> > > >> serialization framework for a long time. The problem ended up being
> > > >> the analysis-paralysis, rather than the specific implementation
> > > >> problem. In other words, the problem was a LACK of implementation
> than
> > > >> actual REAL problems.
> > > >>
> > > >> Based on this experience, I'd strongly suggest adopting protobuf and
> > > >> moving on. Forget about pluggable RPC implementations, the
> complexity
> > > >> doesnt deliver benefits. The benefits of protobuf is that its the
> RPC
> > > >> format for Hadoop and HBase, which allows Drill to draw on the broad
> > > >> experience of those communities who need to implement high
> performance
> > > >> backwards compatible RPC serialization.
> > > >>
> > > >> ====
> > > >>
> > > >> Expanding a bit, I've looked in to this issue a lot, and there is
> very
> > > >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> > > >> percent faster of this, and that, etc.  I'd strongly suggest
> protobuf
> > > >> for the expanded community.  There is no particular Apache
> imperative
> > > >> that Apache projects re-use libraries.  Use what makes sense for
> your
> > > >> project.
> > > >>
> > > >> As regards to Avro, it's a fine serialization format for long term
> > > >> data retention, but the complexities that exist to enable that make
> it
> > > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any
> form.
> > > >>
> > > >> -ryan
> > > >>
> > > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <tshiran@maprtech.com
> >
> > > >> wrote:
> > > >>> We plan to propose the architecture and interfaces in the next
> couple
> > > >>> weeks, which will make it easy to divide the project into clear
> > > building
> > > >>> blocks. At that point it will be easier to start contributing
> > different
> > > >>> data sources, data formats, operators, query languages, etc.
> > > >>>
> > > >>> The contributions are done in the usual Apache way. It's best to
> > open a
> > > >>> JIRA and then post a patch so that others can review and then a
> > > committer
> > > >>> can check it in.
> > > >>>
> > > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > >> chandanmadhesia@gmail.com
> > > >>>> wrote:
> > > >>>
> > > >>>> Hi
> > > >>>>
> > > >>>> Hi
> > > >>>>
> > > >>>> What is the process to become a contributor to drill ?
> > > >>>>
> > > >>>> Regards
> > > >>>> chandan
> > > >>>>
> > > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> ted.dunning@gmail.com>
> > > >> wrote:
> > > >>>>
> > > >>>>> Suffice it to say that if *you* think it is important enough to
> > > >> implement
> > > >>>>> and maintain, then the group shouldn't say naye.  The consensus
> > stuff
> > > >>>>> should only block things that break something else.  Additive
> > > features
> > > >>>> that
> > > >>>>> are highly maintainable (or which come with commitments)
> shouldn't
> > > >>>>> generally be blocked.
> > > >>>>>
> > > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > >>>>> michael.hausenblas@gmail.com> wrote:
> > > >>>>>
> > > >>>>>> Good. Feel free to put me down for that, if the group as a whole
> > > >> thinks
> > > >>>>>> that (supporting Thrift) makes sense.
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Tomer Shiran
> > > >>> Director of Product Management | MapR Technologies | 650-804-8657
> > > >>
> > >
> > >
> >
> >
> > --
> > My research interests are distributed systems, parallel computing and
> > bytecode based virtual machine.
> >
> > My profile:
> > http://www.linkedin.com/in/coderplay
> > My blog:
> > http://coderplay.javaeye.com
> >
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Re: Thrift?

Posted by Hyunsik Choi <hy...@gmail.com>.
+1 for json as initial data format

In addition, I recommend YarnRPC with protocol buffer for internal RPC and
API RPC. Protocol buffer is portable to other languages. If we use another
RPC system, we have to additionally consider the security aspect of Hadoop.

--
Hyunsik Choi

On Sat, Sep 15, 2012 at 8:39 PM, Min Zhou <co...@gmail.com> wrote:

> There should be 2 types of serialization method. One should define its
> schema,
> for the use of RPC,  user wire API; while the other need not define schema,
> it
> typically for internal data transfer, I think fastjson or kryo is quite
> suitable for the
> latter purpose.
>
>
> Regards,
> Min
>
> On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> michael.hausenblas@gmail.com> wrote:
>
> >
> > Point taken … +1 for protobuf - from my POV we can close ISSUE-1
> >
> > > The question of an internal wire format, btw, does not constrain the
> > project relative to external access.
> >
> > Sounds sensible.
> >
> > The only one thing I really don't get is: why did you put Avro and JSON
> > into the proposal [1] in the first place? Or is this the 'external
> access'
> > from above?
> >
> > Cheers,
> >            Michael
> >
> > [1] http://wiki.apache.org/incubator/DrillProposal
> >
> > --
> > Michael Hausenblas
> > Ireland, Europe
> > http://mhausenblas.info/
> >
> > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> >
> > > I think that it is important to ask a few questions leading up a
> decision
> > > here.
> > >
> > > The first is a (rhetorical) show of hands about how many people believe
> > > that there are no serious performance or expressivity killers when
> > > comparing alternative serialization frameworks.  As far as I know,
> > > performance differences are not massive (and protobufs is one of the
> > > leaders in any case) and the expressivity differences are essentially
> > nil.
> > > If somebody feels that there is a serious show-stopper with any option,
> > > they should speak.
> > >
> > > The second is to ask the sense of the community whether they judge
> > progress
> > > or perfection in this decision is most important to the project.  My
> > guess
> > > is that almost everybody would prefer to see progress as long as the
> > > technical choice is not subject to some horrid missing bit.
> > >
> > > The final question is whether it is reasonable to go along with
> protobufs
> > > given that several very experienced engineers prefer it and would like
> to
> > > produce code based on it.  If the first two answers are answered to the
> > > effect of protobufs is about as good as we will find and that progress
> > > trumps small differences, then it seems that moving to follow this
> > > preference of Jason and Ryan for protobufs might be a reasonable thing
> to
> > > do.
> > >
> > > The question of an internal wire format, btw, does not constrain the
> > > project relative to external access.  I think it is important to
> support
> > > JDBC and ODBC and whatever is in common use for querying.  For external
> > > access the question is quite different.  Whereas for the internal
> format
> > > consensus around a single choice has large benefits, the external
> format
> > > choice is nearly the opposite.  For an external format, limiting
> > ourselves
> > > to a single choice seems like a bad idea and increasing the audience
> > seems
> > > like a better choice.
> > >
> > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> I just commented on this first JIRA.  Here is my text:
> > >>
> > >> This issue has been hashed over a lot in the Hadoop projects. There
> > >> was work done to compare thrift vs avro vs protobuf. The conclusion
> > >> was protobuf was the decision to use.
> > >>
> > >> Prior to this move, there had been a lot of noise about pluggable RPC
> > >> transports, and whatnot. It held up adoption of a backwards compatible
> > >> serialization framework for a long time. The problem ended up being
> > >> the analysis-paralysis, rather than the specific implementation
> > >> problem. In other words, the problem was a LACK of implementation than
> > >> actual REAL problems.
> > >>
> > >> Based on this experience, I'd strongly suggest adopting protobuf and
> > >> moving on. Forget about pluggable RPC implementations, the complexity
> > >> doesnt deliver benefits. The benefits of protobuf is that its the RPC
> > >> format for Hadoop and HBase, which allows Drill to draw on the broad
> > >> experience of those communities who need to implement high performance
> > >> backwards compatible RPC serialization.
> > >>
> > >> ====
> > >>
> > >> Expanding a bit, I've looked in to this issue a lot, and there is very
> > >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> > >> percent faster of this, and that, etc.  I'd strongly suggest protobuf
> > >> for the expanded community.  There is no particular Apache imperative
> > >> that Apache projects re-use libraries.  Use what makes sense for your
> > >> project.
> > >>
> > >> As regards to Avro, it's a fine serialization format for long term
> > >> data retention, but the complexities that exist to enable that make it
> > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
> > >>
> > >> -ryan
> > >>
> > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
> > >> wrote:
> > >>> We plan to propose the architecture and interfaces in the next couple
> > >>> weeks, which will make it easy to divide the project into clear
> > building
> > >>> blocks. At that point it will be easier to start contributing
> different
> > >>> data sources, data formats, operators, query languages, etc.
> > >>>
> > >>> The contributions are done in the usual Apache way. It's best to
> open a
> > >>> JIRA and then post a patch so that others can review and then a
> > committer
> > >>> can check it in.
> > >>>
> > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > >> chandanmadhesia@gmail.com
> > >>>> wrote:
> > >>>
> > >>>> Hi
> > >>>>
> > >>>> Hi
> > >>>>
> > >>>> What is the process to become a contributor to drill ?
> > >>>>
> > >>>> Regards
> > >>>> chandan
> > >>>>
> > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
> > >> wrote:
> > >>>>
> > >>>>> Suffice it to say that if *you* think it is important enough to
> > >> implement
> > >>>>> and maintain, then the group shouldn't say naye.  The consensus
> stuff
> > >>>>> should only block things that break something else.  Additive
> > features
> > >>>> that
> > >>>>> are highly maintainable (or which come with commitments) shouldn't
> > >>>>> generally be blocked.
> > >>>>>
> > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > >>>>> michael.hausenblas@gmail.com> wrote:
> > >>>>>
> > >>>>>> Good. Feel free to put me down for that, if the group as a whole
> > >> thinks
> > >>>>>> that (supporting Thrift) makes sense.
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Tomer Shiran
> > >>> Director of Product Management | MapR Technologies | 650-804-8657
> > >>
> >
> >
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

Re: Thrift?

Posted by Min Zhou <co...@gmail.com>.
MessagePack +1 for internal use.

MessagePack is a good candidate. It's under Apache License.

On Sat, Sep 15, 2012 at 7:46 PM, Ted Dunning <te...@gmail.com> wrote:

> I have heard very good words about MessagePack ( http://msgpack.org/ ) as
> a
> binary JSON format.  It supports C++ and Java and is very fast.
>
> Kryo is nice (see Storm for a success story) but it is very much limited to
> Java and your wire protocol depends on you class structure which results in
> a very bad abstraction leak.
>
> Our requirements for the internal API technology include:
>
> - support both C++ and Java
>
> - super fast
>
> - abstracted away from the actual class definition
>
> - other stuff
>
> Kryo fails badly on C++ and abstraction.  It is fabulous as a replacement
> for Java serialization, but just isn't portable.  I love it for what it is,
> but it isn't what we need here.
>
> On Sat, Sep 15, 2012 at 4:39 AM, Min Zhou <co...@gmail.com> wrote:
>
> > There should be 2 types of serialization method. One should define its
> > schema, for the use of RPC,  user wire API; while the other need not
> > define schema, it typically for internal data transfer, I think fastjson
> or
> > kryo is quite
> > suitable for the latter purpose.
> >
> >
> > Regards,
> > Min
> >
> > On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> > michael.hausenblas@gmail.com> wrote:
> >
> > >
> > > Point taken … +1 for protobuf - from my POV we can close ISSUE-1
> > >
> > > > The question of an internal wire format, btw, does not constrain the
> > > project relative to external access.
> > >
> > > Sounds sensible.
> > >
> > > The only one thing I really don't get is: why did you put Avro and JSON
> > > into the proposal [1] in the first place? Or is this the 'external
> > access'
> > > from above?
> > >
> > > Cheers,
> > >            Michael
> > >
> > > [1] http://wiki.apache.org/incubator/DrillProposal
> > >
> > > --
> > > Michael Hausenblas
> > > Ireland, Europe
> > > http://mhausenblas.info/
> > >
> > > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> > >
> > > > I think that it is important to ask a few questions leading up a
> > decision
> > > > here.
> > > >
> > > > The first is a (rhetorical) show of hands about how many people
> believe
> > > > that there are no serious performance or expressivity killers when
> > > > comparing alternative serialization frameworks.  As far as I know,
> > > > performance differences are not massive (and protobufs is one of the
> > > > leaders in any case) and the expressivity differences are essentially
> > > nil.
> > > > If somebody feels that there is a serious show-stopper with any
> option,
> > > > they should speak.
> > > >
> > > > The second is to ask the sense of the community whether they judge
> > > progress
> > > > or perfection in this decision is most important to the project.  My
> > > guess
> > > > is that almost everybody would prefer to see progress as long as the
> > > > technical choice is not subject to some horrid missing bit.
> > > >
> > > > The final question is whether it is reasonable to go along with
> > protobufs
> > > > given that several very experienced engineers prefer it and would
> like
> > to
> > > > produce code based on it.  If the first two answers are answered to
> the
> > > > effect of protobufs is about as good as we will find and that
> progress
> > > > trumps small differences, then it seems that moving to follow this
> > > > preference of Jason and Ryan for protobufs might be a reasonable
> thing
> > to
> > > > do.
> > > >
> > > > The question of an internal wire format, btw, does not constrain the
> > > > project relative to external access.  I think it is important to
> > support
> > > > JDBC and ODBC and whatever is in common use for querying.  For
> external
> > > > access the question is quite different.  Whereas for the internal
> > format
> > > > consensus around a single choice has large benefits, the external
> > format
> > > > choice is nearly the opposite.  For an external format, limiting
> > > ourselves
> > > > to a single choice seems like a bad idea and increasing the audience
> > > seems
> > > > like a better choice.
> > > >
> > > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> I just commented on this first JIRA.  Here is my text:
> > > >>
> > > >> This issue has been hashed over a lot in the Hadoop projects. There
> > > >> was work done to compare thrift vs avro vs protobuf. The conclusion
> > > >> was protobuf was the decision to use.
> > > >>
> > > >> Prior to this move, there had been a lot of noise about pluggable
> RPC
> > > >> transports, and whatnot. It held up adoption of a backwards
> compatible
> > > >> serialization framework for a long time. The problem ended up being
> > > >> the analysis-paralysis, rather than the specific implementation
> > > >> problem. In other words, the problem was a LACK of implementation
> than
> > > >> actual REAL problems.
> > > >>
> > > >> Based on this experience, I'd strongly suggest adopting protobuf and
> > > >> moving on. Forget about pluggable RPC implementations, the
> complexity
> > > >> doesnt deliver benefits. The benefits of protobuf is that its the
> RPC
> > > >> format for Hadoop and HBase, which allows Drill to draw on the broad
> > > >> experience of those communities who need to implement high
> performance
> > > >> backwards compatible RPC serialization.
> > > >>
> > > >> ====
> > > >>
> > > >> Expanding a bit, I've looked in to this issue a lot, and there is
> very
> > > >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> > > >> percent faster of this, and that, etc.  I'd strongly suggest
> protobuf
> > > >> for the expanded community.  There is no particular Apache
> imperative
> > > >> that Apache projects re-use libraries.  Use what makes sense for
> your
> > > >> project.
> > > >>
> > > >> As regards to Avro, it's a fine serialization format for long term
> > > >> data retention, but the complexities that exist to enable that make
> it
> > > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any
> form.
> > > >>
> > > >> -ryan
> > > >>
> > > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <tshiran@maprtech.com
> >
> > > >> wrote:
> > > >>> We plan to propose the architecture and interfaces in the next
> couple
> > > >>> weeks, which will make it easy to divide the project into clear
> > > building
> > > >>> blocks. At that point it will be easier to start contributing
> > different
> > > >>> data sources, data formats, operators, query languages, etc.
> > > >>>
> > > >>> The contributions are done in the usual Apache way. It's best to
> > open a
> > > >>> JIRA and then post a patch so that others can review and then a
> > > committer
> > > >>> can check it in.
> > > >>>
> > > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > >> chandanmadhesia@gmail.com
> > > >>>> wrote:
> > > >>>
> > > >>>> Hi
> > > >>>>
> > > >>>> Hi
> > > >>>>
> > > >>>> What is the process to become a contributor to drill ?
> > > >>>>
> > > >>>> Regards
> > > >>>> chandan
> > > >>>>
> > > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> ted.dunning@gmail.com>
> > > >> wrote:
> > > >>>>
> > > >>>>> Suffice it to say that if *you* think it is important enough to
> > > >> implement
> > > >>>>> and maintain, then the group shouldn't say naye.  The consensus
> > stuff
> > > >>>>> should only block things that break something else.  Additive
> > > features
> > > >>>> that
> > > >>>>> are highly maintainable (or which come with commitments)
> shouldn't
> > > >>>>> generally be blocked.
> > > >>>>>
> > > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > >>>>> michael.hausenblas@gmail.com> wrote:
> > > >>>>>
> > > >>>>>> Good. Feel free to put me down for that, if the group as a whole
> > > >> thinks
> > > >>>>>> that (supporting Thrift) makes sense.
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Tomer Shiran
> > > >>> Director of Product Management | MapR Technologies | 650-804-8657
> > > >>
> > >
> > >
> >
> >
> > --
> > My research interests are distributed systems, parallel computing and
> > bytecode based virtual machine.
> >
> > My profile:
> > http://www.linkedin.com/in/coderplay
> > My blog:
> > http://coderplay.javaeye.com
> >
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
I have heard very good words about MessagePack ( http://msgpack.org/ ) as a
binary JSON format.  It supports C++ and Java and is very fast.

Kryo is nice (see Storm for a success story) but it is very much limited to
Java and your wire protocol depends on you class structure which results in
a very bad abstraction leak.

Our requirements for the internal API technology include:

- support both C++ and Java

- super fast

- abstracted away from the actual class definition

- other stuff

Kryo fails badly on C++ and abstraction.  It is fabulous as a replacement
for Java serialization, but just isn't portable.  I love it for what it is,
but it isn't what we need here.

On Sat, Sep 15, 2012 at 4:39 AM, Min Zhou <co...@gmail.com> wrote:

> There should be 2 types of serialization method. One should define its
> schema, for the use of RPC,  user wire API; while the other need not
> define schema, it typically for internal data transfer, I think fastjson or
> kryo is quite
> suitable for the latter purpose.
>
>
> Regards,
> Min
>
> On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
> michael.hausenblas@gmail.com> wrote:
>
> >
> > Point taken … +1 for protobuf - from my POV we can close ISSUE-1
> >
> > > The question of an internal wire format, btw, does not constrain the
> > project relative to external access.
> >
> > Sounds sensible.
> >
> > The only one thing I really don't get is: why did you put Avro and JSON
> > into the proposal [1] in the first place? Or is this the 'external
> access'
> > from above?
> >
> > Cheers,
> >            Michael
> >
> > [1] http://wiki.apache.org/incubator/DrillProposal
> >
> > --
> > Michael Hausenblas
> > Ireland, Europe
> > http://mhausenblas.info/
> >
> > On 14 Sep 2012, at 22:31, Ted Dunning wrote:
> >
> > > I think that it is important to ask a few questions leading up a
> decision
> > > here.
> > >
> > > The first is a (rhetorical) show of hands about how many people believe
> > > that there are no serious performance or expressivity killers when
> > > comparing alternative serialization frameworks.  As far as I know,
> > > performance differences are not massive (and protobufs is one of the
> > > leaders in any case) and the expressivity differences are essentially
> > nil.
> > > If somebody feels that there is a serious show-stopper with any option,
> > > they should speak.
> > >
> > > The second is to ask the sense of the community whether they judge
> > progress
> > > or perfection in this decision is most important to the project.  My
> > guess
> > > is that almost everybody would prefer to see progress as long as the
> > > technical choice is not subject to some horrid missing bit.
> > >
> > > The final question is whether it is reasonable to go along with
> protobufs
> > > given that several very experienced engineers prefer it and would like
> to
> > > produce code based on it.  If the first two answers are answered to the
> > > effect of protobufs is about as good as we will find and that progress
> > > trumps small differences, then it seems that moving to follow this
> > > preference of Jason and Ryan for protobufs might be a reasonable thing
> to
> > > do.
> > >
> > > The question of an internal wire format, btw, does not constrain the
> > > project relative to external access.  I think it is important to
> support
> > > JDBC and ODBC and whatever is in common use for querying.  For external
> > > access the question is quite different.  Whereas for the internal
> format
> > > consensus around a single choice has large benefits, the external
> format
> > > choice is nearly the opposite.  For an external format, limiting
> > ourselves
> > > to a single choice seems like a bad idea and increasing the audience
> > seems
> > > like a better choice.
> > >
> > > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> I just commented on this first JIRA.  Here is my text:
> > >>
> > >> This issue has been hashed over a lot in the Hadoop projects. There
> > >> was work done to compare thrift vs avro vs protobuf. The conclusion
> > >> was protobuf was the decision to use.
> > >>
> > >> Prior to this move, there had been a lot of noise about pluggable RPC
> > >> transports, and whatnot. It held up adoption of a backwards compatible
> > >> serialization framework for a long time. The problem ended up being
> > >> the analysis-paralysis, rather than the specific implementation
> > >> problem. In other words, the problem was a LACK of implementation than
> > >> actual REAL problems.
> > >>
> > >> Based on this experience, I'd strongly suggest adopting protobuf and
> > >> moving on. Forget about pluggable RPC implementations, the complexity
> > >> doesnt deliver benefits. The benefits of protobuf is that its the RPC
> > >> format for Hadoop and HBase, which allows Drill to draw on the broad
> > >> experience of those communities who need to implement high performance
> > >> backwards compatible RPC serialization.
> > >>
> > >> ====
> > >>
> > >> Expanding a bit, I've looked in to this issue a lot, and there is very
> > >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> > >> percent faster of this, and that, etc.  I'd strongly suggest protobuf
> > >> for the expanded community.  There is no particular Apache imperative
> > >> that Apache projects re-use libraries.  Use what makes sense for your
> > >> project.
> > >>
> > >> As regards to Avro, it's a fine serialization format for long term
> > >> data retention, but the complexities that exist to enable that make it
> > >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
> > >>
> > >> -ryan
> > >>
> > >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
> > >> wrote:
> > >>> We plan to propose the architecture and interfaces in the next couple
> > >>> weeks, which will make it easy to divide the project into clear
> > building
> > >>> blocks. At that point it will be easier to start contributing
> different
> > >>> data sources, data formats, operators, query languages, etc.
> > >>>
> > >>> The contributions are done in the usual Apache way. It's best to
> open a
> > >>> JIRA and then post a patch so that others can review and then a
> > committer
> > >>> can check it in.
> > >>>
> > >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > >> chandanmadhesia@gmail.com
> > >>>> wrote:
> > >>>
> > >>>> Hi
> > >>>>
> > >>>> Hi
> > >>>>
> > >>>> What is the process to become a contributor to drill ?
> > >>>>
> > >>>> Regards
> > >>>> chandan
> > >>>>
> > >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
> > >> wrote:
> > >>>>
> > >>>>> Suffice it to say that if *you* think it is important enough to
> > >> implement
> > >>>>> and maintain, then the group shouldn't say naye.  The consensus
> stuff
> > >>>>> should only block things that break something else.  Additive
> > features
> > >>>> that
> > >>>>> are highly maintainable (or which come with commitments) shouldn't
> > >>>>> generally be blocked.
> > >>>>>
> > >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > >>>>> michael.hausenblas@gmail.com> wrote:
> > >>>>>
> > >>>>>> Good. Feel free to put me down for that, if the group as a whole
> > >> thinks
> > >>>>>> that (supporting Thrift) makes sense.
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Tomer Shiran
> > >>> Director of Product Management | MapR Technologies | 650-804-8657
> > >>
> >
> >
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

Re: Thrift?

Posted by Min Zhou <co...@gmail.com>.
There should be 2 types of serialization method. One should define its
schema,
for the use of RPC,  user wire API; while the other need not define schema,
it
typically for internal data transfer, I think fastjson or kryo is quite
suitable for the
latter purpose.


Regards,
Min

On Sat, Sep 15, 2012 at 5:49 PM, Michael Hausenblas <
michael.hausenblas@gmail.com> wrote:

>
> Point taken … +1 for protobuf - from my POV we can close ISSUE-1
>
> > The question of an internal wire format, btw, does not constrain the
> project relative to external access.
>
> Sounds sensible.
>
> The only one thing I really don't get is: why did you put Avro and JSON
> into the proposal [1] in the first place? Or is this the 'external access'
> from above?
>
> Cheers,
>            Michael
>
> [1] http://wiki.apache.org/incubator/DrillProposal
>
> --
> Michael Hausenblas
> Ireland, Europe
> http://mhausenblas.info/
>
> On 14 Sep 2012, at 22:31, Ted Dunning wrote:
>
> > I think that it is important to ask a few questions leading up a decision
> > here.
> >
> > The first is a (rhetorical) show of hands about how many people believe
> > that there are no serious performance or expressivity killers when
> > comparing alternative serialization frameworks.  As far as I know,
> > performance differences are not massive (and protobufs is one of the
> > leaders in any case) and the expressivity differences are essentially
> nil.
> > If somebody feels that there is a serious show-stopper with any option,
> > they should speak.
> >
> > The second is to ask the sense of the community whether they judge
> progress
> > or perfection in this decision is most important to the project.  My
> guess
> > is that almost everybody would prefer to see progress as long as the
> > technical choice is not subject to some horrid missing bit.
> >
> > The final question is whether it is reasonable to go along with protobufs
> > given that several very experienced engineers prefer it and would like to
> > produce code based on it.  If the first two answers are answered to the
> > effect of protobufs is about as good as we will find and that progress
> > trumps small differences, then it seems that moving to follow this
> > preference of Jason and Ryan for protobufs might be a reasonable thing to
> > do.
> >
> > The question of an internal wire format, btw, does not constrain the
> > project relative to external access.  I think it is important to support
> > JDBC and ODBC and whatever is in common use for querying.  For external
> > access the question is quite different.  Whereas for the internal format
> > consensus around a single choice has large benefits, the external format
> > choice is nearly the opposite.  For an external format, limiting
> ourselves
> > to a single choice seems like a bad idea and increasing the audience
> seems
> > like a better choice.
> >
> > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> wrote:
> >
> >> Hi folks,
> >>
> >> I just commented on this first JIRA.  Here is my text:
> >>
> >> This issue has been hashed over a lot in the Hadoop projects. There
> >> was work done to compare thrift vs avro vs protobuf. The conclusion
> >> was protobuf was the decision to use.
> >>
> >> Prior to this move, there had been a lot of noise about pluggable RPC
> >> transports, and whatnot. It held up adoption of a backwards compatible
> >> serialization framework for a long time. The problem ended up being
> >> the analysis-paralysis, rather than the specific implementation
> >> problem. In other words, the problem was a LACK of implementation than
> >> actual REAL problems.
> >>
> >> Based on this experience, I'd strongly suggest adopting protobuf and
> >> moving on. Forget about pluggable RPC implementations, the complexity
> >> doesnt deliver benefits. The benefits of protobuf is that its the RPC
> >> format for Hadoop and HBase, which allows Drill to draw on the broad
> >> experience of those communities who need to implement high performance
> >> backwards compatible RPC serialization.
> >>
> >> ====
> >>
> >> Expanding a bit, I've looked in to this issue a lot, and there is very
> >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> >> percent faster of this, and that, etc.  I'd strongly suggest protobuf
> >> for the expanded community.  There is no particular Apache imperative
> >> that Apache projects re-use libraries.  Use what makes sense for your
> >> project.
> >>
> >> As regards to Avro, it's a fine serialization format for long term
> >> data retention, but the complexities that exist to enable that make it
> >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
> >>
> >> -ryan
> >>
> >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
> >> wrote:
> >>> We plan to propose the architecture and interfaces in the next couple
> >>> weeks, which will make it easy to divide the project into clear
> building
> >>> blocks. At that point it will be easier to start contributing different
> >>> data sources, data formats, operators, query languages, etc.
> >>>
> >>> The contributions are done in the usual Apache way. It's best to open a
> >>> JIRA and then post a patch so that others can review and then a
> committer
> >>> can check it in.
> >>>
> >>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> >> chandanmadhesia@gmail.com
> >>>> wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> Hi
> >>>>
> >>>> What is the process to become a contributor to drill ?
> >>>>
> >>>> Regards
> >>>> chandan
> >>>>
> >>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
> >> wrote:
> >>>>
> >>>>> Suffice it to say that if *you* think it is important enough to
> >> implement
> >>>>> and maintain, then the group shouldn't say naye.  The consensus stuff
> >>>>> should only block things that break something else.  Additive
> features
> >>>> that
> >>>>> are highly maintainable (or which come with commitments) shouldn't
> >>>>> generally be blocked.
> >>>>>
> >>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> >>>>> michael.hausenblas@gmail.com> wrote:
> >>>>>
> >>>>>> Good. Feel free to put me down for that, if the group as a whole
> >> thinks
> >>>>>> that (supporting Thrift) makes sense.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Tomer Shiran
> >>> Director of Product Management | MapR Technologies | 650-804-8657
> >>
>
>


-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Re: Thrift?

Posted by Michael Hausenblas <mi...@gmail.com>.
Point taken … +1 for protobuf - from my POV we can close ISSUE-1

> The question of an internal wire format, btw, does not constrain the project relative to external access.

Sounds sensible.

The only one thing I really don't get is: why did you put Avro and JSON into the proposal [1] in the first place? Or is this the 'external access' from above?

Cheers,
	   Michael

[1] http://wiki.apache.org/incubator/DrillProposal

--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/

On 14 Sep 2012, at 22:31, Ted Dunning wrote:

> I think that it is important to ask a few questions leading up a decision
> here.
> 
> The first is a (rhetorical) show of hands about how many people believe
> that there are no serious performance or expressivity killers when
> comparing alternative serialization frameworks.  As far as I know,
> performance differences are not massive (and protobufs is one of the
> leaders in any case) and the expressivity differences are essentially nil.
> If somebody feels that there is a serious show-stopper with any option,
> they should speak.
> 
> The second is to ask the sense of the community whether they judge progress
> or perfection in this decision is most important to the project.  My guess
> is that almost everybody would prefer to see progress as long as the
> technical choice is not subject to some horrid missing bit.
> 
> The final question is whether it is reasonable to go along with protobufs
> given that several very experienced engineers prefer it and would like to
> produce code based on it.  If the first two answers are answered to the
> effect of protobufs is about as good as we will find and that progress
> trumps small differences, then it seems that moving to follow this
> preference of Jason and Ryan for protobufs might be a reasonable thing to
> do.
> 
> The question of an internal wire format, btw, does not constrain the
> project relative to external access.  I think it is important to support
> JDBC and ODBC and whatever is in common use for querying.  For external
> access the question is quite different.  Whereas for the internal format
> consensus around a single choice has large benefits, the external format
> choice is nearly the opposite.  For an external format, limiting ourselves
> to a single choice seems like a bad idea and increasing the audience seems
> like a better choice.
> 
> On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com> wrote:
> 
>> Hi folks,
>> 
>> I just commented on this first JIRA.  Here is my text:
>> 
>> This issue has been hashed over a lot in the Hadoop projects. There
>> was work done to compare thrift vs avro vs protobuf. The conclusion
>> was protobuf was the decision to use.
>> 
>> Prior to this move, there had been a lot of noise about pluggable RPC
>> transports, and whatnot. It held up adoption of a backwards compatible
>> serialization framework for a long time. The problem ended up being
>> the analysis-paralysis, rather than the specific implementation
>> problem. In other words, the problem was a LACK of implementation than
>> actual REAL problems.
>> 
>> Based on this experience, I'd strongly suggest adopting protobuf and
>> moving on. Forget about pluggable RPC implementations, the complexity
>> doesnt deliver benefits. The benefits of protobuf is that its the RPC
>> format for Hadoop and HBase, which allows Drill to draw on the broad
>> experience of those communities who need to implement high performance
>> backwards compatible RPC serialization.
>> 
>> ====
>> 
>> Expanding a bit, I've looked in to this issue a lot, and there is very
>> few significant concrete reasons to choose protobuf vs thrift.  Tiny
>> percent faster of this, and that, etc.  I'd strongly suggest protobuf
>> for the expanded community.  There is no particular Apache imperative
>> that Apache projects re-use libraries.  Use what makes sense for your
>> project.
>> 
>> As regards to Avro, it's a fine serialization format for long term
>> data retention, but the complexities that exist to enable that make it
>> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
>> 
>> -ryan
>> 
>> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
>> wrote:
>>> We plan to propose the architecture and interfaces in the next couple
>>> weeks, which will make it easy to divide the project into clear building
>>> blocks. At that point it will be easier to start contributing different
>>> data sources, data formats, operators, query languages, etc.
>>> 
>>> The contributions are done in the usual Apache way. It's best to open a
>>> JIRA and then post a patch so that others can review and then a committer
>>> can check it in.
>>> 
>>> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
>> chandanmadhesia@gmail.com
>>>> wrote:
>>> 
>>>> Hi
>>>> 
>>>> Hi
>>>> 
>>>> What is the process to become a contributor to drill ?
>>>> 
>>>> Regards
>>>> chandan
>>>> 
>>>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>>>> 
>>>>> Suffice it to say that if *you* think it is important enough to
>> implement
>>>>> and maintain, then the group shouldn't say naye.  The consensus stuff
>>>>> should only block things that break something else.  Additive features
>>>> that
>>>>> are highly maintainable (or which come with commitments) shouldn't
>>>>> generally be blocked.
>>>>> 
>>>>> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
>>>>> michael.hausenblas@gmail.com> wrote:
>>>>> 
>>>>>> Good. Feel free to put me down for that, if the group as a whole
>> thinks
>>>>>> that (supporting Thrift) makes sense.
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Tomer Shiran
>>> Director of Product Management | MapR Technologies | 650-804-8657
>> 


Re: Thrift?

Posted by Min Zhou <co...@gmail.com>.
Protobuf +1

Even though the google hasn't officially open source their internal RPC
based on protobuf, there are lots of third-party implementations for all
most all of the popular languages. After all, protobuf give me a better
use experience then that of thrift :)

Thanks,
Min


On Sat, Sep 15, 2012 at 9:56 AM, Clark Yang (杨卓荦) <ya...@gmail.com>wrote:

> protobuf +1
> I don't think it is a standard problem. protobuf has already shown a great
> many benefits and success in many open source projects. It's widely used
> and few better alternative, I think.
>
> BTW, I have posted the first comment of the first jira.
>
> Cheers,
> Zhuoluo (Clark) Yang
>
>
>
> 2012/9/15 Constantine Peresypkin <pc...@gmail.com>
>
> > I really have no idea how one can estimate telco traffic.
> > But I highly doubt that you can fruitfully compare reliability of
> > internal-only protocol (same implementation, easy to enforce
> compatibility)
> > to an interoperable one.
> >
> > On Sat, Sep 15, 2012 at 1:41 AM, Ryan Rawson <ry...@gmail.com> wrote:
> >
> > > I didn't say I was the one making the argument...
> > >
> > > Google has put probably > 10^24 bytes of data thru protobuf in
> > > multiple implementations (eg: serialization on disk and on wire RPC).
> > >   That is a low estimate.
> > >
> > > I'd be interested in hearing what 20 years of telco protocol traffic
> > > might compare to 10 years of google's usage of protobuf.  Exponential
> > > curve and all of that.
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Sep 14, 2012 at 3:36 PM, Constantine Peresypkin
> > > <pc...@gmail.com> wrote:
> > > > More battle tested than more than 20 year old standard used almost in
> > > every
> > > > telecom protocol that exists nowdays?
> > > > I think your statement is a little on "too bold" side. :)
> > > >
> > > > On Sat, Sep 15, 2012 at 1:30 AM, Ryan Rawson <ry...@gmail.com>
> > wrote:
> > > >
> > > >> Funny thing, given how much use protobufs has been put thru, I think
> > > >> one could make the argument its more battle tested than ASN.1 ...
> > > >>
> > > >> On Fri, Sep 14, 2012 at 3:24 PM, Constantine Peresypkin
> > > >> <co...@litestack.com> wrote:
> > > >> > Protobuf is an attempt to make ASN.1 more developer friendly (not
> a
> > > bad
> > > >> > attempt).
> > > >> > It's simpler, has much less features, easier to implement and has
> a
> > > >> compact
> > > >> > encoding.
> > > >> > But on other hand it's non-standard, "reinvented wheel" they could
> > > just
> > > >> do
> > > >> > a "better than PER" encoding for ASN.1, and AFAIK has no support
> for
> > > the
> > > >> > new and shiny Google encodings, like "group varint".
> > > >> > All in all in current situation it seems a better choice than
> ASN.1,
> > > not
> > > >> > even arguing about something even more vague and non-standard as
> > > Thrift.
> > > >> >
> > > >> > On Sat, Sep 15, 2012 at 12:38 AM, Ryan Rawson <ryanobjc@gmail.com
> >
> > > >> wrote:
> > > >> >
> > > >> >> Thanks for that Ted.
> > > >> >>
> > > >> >> Correct - internal wire format doesnt mean 'drill only supports
> > > >> >> protobuf encoded data'.
> > > >> >>
> > > >> >> Part of the reason to favor protobuf is that a lot of people in
> the
> > > >> >> broader 'big data' community are building a lot of experience
> with
> > > it.
> > > >> >>  Hadoop and HBase both are moving to/moved to protobuf on the
> wire.
> > > >> >> Being able to leverage this expertise is valuable.
> > > >> >>
> > > >> >> There is a JIRA in Hadoop-land where someone had done a deep dive
> > > >> >> 'bake off' between thrift, protobuf and avro.  The ultimate
> choice
> > > was
> > > >> >> protobuf for a number of reasons.  If people want to re-do the
> > > >> >> analysis, I'd like to see it in the context of THAT analysis (eg:
> > why
> > > >> >> the assumptions there are not the same for Drill)... if anything
> > it'd
> > > >> >> give a concrete form to what can be a mire.
> > > >> >>
> > > >> >> For what it's worth, I've had many discussion along these angles
> > with
> > > >> >> a variety of people including committers on Thrift, and the
> > consensus
> > > >> >> is both are good choices.
> > > >> >>
> > > >> >> -ryan
> > > >> >>
> > > >> >> On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <
> > ted.dunning@gmail.com>
> > > >> >> wrote:
> > > >> >> > I think that it is important to ask a few questions leading up
> a
> > > >> decision
> > > >> >> > here.
> > > >> >> >
> > > >> >> > The first is a (rhetorical) show of hands about how many people
> > > >> believe
> > > >> >> > that there are no serious performance or expressivity killers
> > when
> > > >> >> > comparing alternative serialization frameworks.  As far as I
> > know,
> > > >> >> > performance differences are not massive (and protobufs is one
> of
> > > the
> > > >> >> > leaders in any case) and the expressivity differences are
> > > essentially
> > > >> >> nil.
> > > >> >> >  If somebody feels that there is a serious show-stopper with
> any
> > > >> option,
> > > >> >> > they should speak.
> > > >> >> >
> > > >> >> > The second is to ask the sense of the community whether they
> > judge
> > > >> >> progress
> > > >> >> > or perfection in this decision is most important to the
> project.
> > >  My
> > > >> >> guess
> > > >> >> > is that almost everybody would prefer to see progress as long
> as
> > > the
> > > >> >> > technical choice is not subject to some horrid missing bit.
> > > >> >> >
> > > >> >> > The final question is whether it is reasonable to go along with
> > > >> protobufs
> > > >> >> > given that several very experienced engineers prefer it and
> would
> > > >> like to
> > > >> >> > produce code based on it.  If the first two answers are
> answered
> > to
> > > >> the
> > > >> >> > effect of protobufs is about as good as we will find and that
> > > progress
> > > >> >> > trumps small differences, then it seems that moving to follow
> > this
> > > >> >> > preference of Jason and Ryan for protobufs might be a
> reasonable
> > > >> thing to
> > > >> >> > do.
> > > >> >> >
> > > >> >> > The question of an internal wire format, btw, does not
> constrain
> > > the
> > > >> >> > project relative to external access.  I think it is important
> to
> > > >> support
> > > >> >> > JDBC and ODBC and whatever is in common use for querying.  For
> > > >> external
> > > >> >> > access the question is quite different.  Whereas for the
> internal
> > > >> format
> > > >> >> > consensus around a single choice has large benefits, the
> external
> > > >> format
> > > >> >> > choice is nearly the opposite.  For an external format,
> limiting
> > > >> >> ourselves
> > > >> >> > to a single choice seems like a bad idea and increasing the
> > > audience
> > > >> >> seems
> > > >> >> > like a better choice.
> > > >> >> >
> > > >> >> > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <
> > ryanobjc@gmail.com>
> > > >> >> wrote:
> > > >> >> >
> > > >> >> >> Hi folks,
> > > >> >> >>
> > > >> >> >> I just commented on this first JIRA.  Here is my text:
> > > >> >> >>
> > > >> >> >> This issue has been hashed over a lot in the Hadoop projects.
> > > There
> > > >> >> >> was work done to compare thrift vs avro vs protobuf. The
> > > conclusion
> > > >> >> >> was protobuf was the decision to use.
> > > >> >> >>
> > > >> >> >> Prior to this move, there had been a lot of noise about
> > pluggable
> > > RPC
> > > >> >> >> transports, and whatnot. It held up adoption of a backwards
> > > >> compatible
> > > >> >> >> serialization framework for a long time. The problem ended up
> > > being
> > > >> >> >> the analysis-paralysis, rather than the specific
> implementation
> > > >> >> >> problem. In other words, the problem was a LACK of
> > implementation
> > > >> than
> > > >> >> >> actual REAL problems.
> > > >> >> >>
> > > >> >> >> Based on this experience, I'd strongly suggest adopting
> protobuf
> > > and
> > > >> >> >> moving on. Forget about pluggable RPC implementations, the
> > > complexity
> > > >> >> >> doesnt deliver benefits. The benefits of protobuf is that its
> > the
> > > RPC
> > > >> >> >> format for Hadoop and HBase, which allows Drill to draw on the
> > > broad
> > > >> >> >> experience of those communities who need to implement high
> > > >> performance
> > > >> >> >> backwards compatible RPC serialization.
> > > >> >> >>
> > > >> >> >> ====
> > > >> >> >>
> > > >> >> >> Expanding a bit, I've looked in to this issue a lot, and there
> > is
> > > >> very
> > > >> >> >> few significant concrete reasons to choose protobuf vs thrift.
> > >  Tiny
> > > >> >> >> percent faster of this, and that, etc.  I'd strongly suggest
> > > protobuf
> > > >> >> >> for the expanded community.  There is no particular Apache
> > > imperative
> > > >> >> >> that Apache projects re-use libraries.  Use what makes sense
> for
> > > your
> > > >> >> >> project.
> > > >> >> >>
> > > >> >> >> As regards to Avro, it's a fine serialization format for long
> > term
> > > >> >> >> data retention, but the complexities that exist to enable that
> > > make
> > > >> it
> > > >> >> >> non-ideal for an RPC.  I know of no one who uses AvroRPC in
> any
> > > form.
> > > >> >> >>
> > > >> >> >> -ryan
> > > >> >> >>
> > > >> >> >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> > > tshiran@maprtech.com>
> > > >> >> >> wrote:
> > > >> >> >> > We plan to propose the architecture and interfaces in the
> next
> > > >> couple
> > > >> >> >> > weeks, which will make it easy to divide the project into
> > clear
> > > >> >> building
> > > >> >> >> > blocks. At that point it will be easier to start
> contributing
> > > >> >> different
> > > >> >> >> > data sources, data formats, operators, query languages, etc.
> > > >> >> >> >
> > > >> >> >> > The contributions are done in the usual Apache way. It's
> best
> > to
> > > >> open
> > > >> >> a
> > > >> >> >> > JIRA and then post a patch so that others can review and
> then
> > a
> > > >> >> committer
> > > >> >> >> > can check it in.
> > > >> >> >> >
> > > >> >> >> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > > >> >> >> chandanmadhesia@gmail.com
> > > >> >> >> >> wrote:
> > > >> >> >> >
> > > >> >> >> >> Hi
> > > >> >> >> >>
> > > >> >> >> >> Hi
> > > >> >> >> >>
> > > >> >> >> >> What is the process to become a contributor to drill ?
> > > >> >> >> >>
> > > >> >> >> >> Regards
> > > >> >> >> >> chandan
> > > >> >> >> >>
> > > >> >> >> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> > > >> ted.dunning@gmail.com>
> > > >> >> >> wrote:
> > > >> >> >> >>
> > > >> >> >> >> > Suffice it to say that if *you* think it is important
> > enough
> > > to
> > > >> >> >> implement
> > > >> >> >> >> > and maintain, then the group shouldn't say naye.  The
> > > consensus
> > > >> >> stuff
> > > >> >> >> >> > should only block things that break something else.
> >  Additive
> > > >> >> features
> > > >> >> >> >> that
> > > >> >> >> >> > are highly maintainable (or which come with commitments)
> > > >> shouldn't
> > > >> >> >> >> > generally be blocked.
> > > >> >> >> >> >
> > > >> >> >> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > > >> >> >> >> > michael.hausenblas@gmail.com> wrote:
> > > >> >> >> >> >
> > > >> >> >> >> > > Good. Feel free to put me down for that, if the group
> as
> > a
> > > >> whole
> > > >> >> >> thinks
> > > >> >> >> >> > > that (supporting Thrift) makes sense.
> > > >> >> >> >> > >
> > > >> >> >> >> >
> > > >> >> >> >>
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > --
> > > >> >> >> > Tomer Shiran
> > > >> >> >> > Director of Product Management | MapR Technologies |
> > > 650-804-8657
> > > >> >> >>
> > > >> >>
> > > >>
> > >
> >
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Re: Thrift?

Posted by "Clark Yang (杨卓荦)" <ya...@gmail.com>.
protobuf +1
I don't think it is a standard problem. protobuf has already shown a great
many benefits and success in many open source projects. It's widely used
and few better alternative, I think.

BTW, I have posted the first comment of the first jira.

Cheers,
Zhuoluo (Clark) Yang



2012/9/15 Constantine Peresypkin <pc...@gmail.com>

> I really have no idea how one can estimate telco traffic.
> But I highly doubt that you can fruitfully compare reliability of
> internal-only protocol (same implementation, easy to enforce compatibility)
> to an interoperable one.
>
> On Sat, Sep 15, 2012 at 1:41 AM, Ryan Rawson <ry...@gmail.com> wrote:
>
> > I didn't say I was the one making the argument...
> >
> > Google has put probably > 10^24 bytes of data thru protobuf in
> > multiple implementations (eg: serialization on disk and on wire RPC).
> >   That is a low estimate.
> >
> > I'd be interested in hearing what 20 years of telco protocol traffic
> > might compare to 10 years of google's usage of protobuf.  Exponential
> > curve and all of that.
> >
> >
> >
> >
> >
> > On Fri, Sep 14, 2012 at 3:36 PM, Constantine Peresypkin
> > <pc...@gmail.com> wrote:
> > > More battle tested than more than 20 year old standard used almost in
> > every
> > > telecom protocol that exists nowdays?
> > > I think your statement is a little on "too bold" side. :)
> > >
> > > On Sat, Sep 15, 2012 at 1:30 AM, Ryan Rawson <ry...@gmail.com>
> wrote:
> > >
> > >> Funny thing, given how much use protobufs has been put thru, I think
> > >> one could make the argument its more battle tested than ASN.1 ...
> > >>
> > >> On Fri, Sep 14, 2012 at 3:24 PM, Constantine Peresypkin
> > >> <co...@litestack.com> wrote:
> > >> > Protobuf is an attempt to make ASN.1 more developer friendly (not a
> > bad
> > >> > attempt).
> > >> > It's simpler, has much less features, easier to implement and has a
> > >> compact
> > >> > encoding.
> > >> > But on other hand it's non-standard, "reinvented wheel" they could
> > just
> > >> do
> > >> > a "better than PER" encoding for ASN.1, and AFAIK has no support for
> > the
> > >> > new and shiny Google encodings, like "group varint".
> > >> > All in all in current situation it seems a better choice than ASN.1,
> > not
> > >> > even arguing about something even more vague and non-standard as
> > Thrift.
> > >> >
> > >> > On Sat, Sep 15, 2012 at 12:38 AM, Ryan Rawson <ry...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> Thanks for that Ted.
> > >> >>
> > >> >> Correct - internal wire format doesnt mean 'drill only supports
> > >> >> protobuf encoded data'.
> > >> >>
> > >> >> Part of the reason to favor protobuf is that a lot of people in the
> > >> >> broader 'big data' community are building a lot of experience with
> > it.
> > >> >>  Hadoop and HBase both are moving to/moved to protobuf on the wire.
> > >> >> Being able to leverage this expertise is valuable.
> > >> >>
> > >> >> There is a JIRA in Hadoop-land where someone had done a deep dive
> > >> >> 'bake off' between thrift, protobuf and avro.  The ultimate choice
> > was
> > >> >> protobuf for a number of reasons.  If people want to re-do the
> > >> >> analysis, I'd like to see it in the context of THAT analysis (eg:
> why
> > >> >> the assumptions there are not the same for Drill)... if anything
> it'd
> > >> >> give a concrete form to what can be a mire.
> > >> >>
> > >> >> For what it's worth, I've had many discussion along these angles
> with
> > >> >> a variety of people including committers on Thrift, and the
> consensus
> > >> >> is both are good choices.
> > >> >>
> > >> >> -ryan
> > >> >>
> > >> >> On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <
> ted.dunning@gmail.com>
> > >> >> wrote:
> > >> >> > I think that it is important to ask a few questions leading up a
> > >> decision
> > >> >> > here.
> > >> >> >
> > >> >> > The first is a (rhetorical) show of hands about how many people
> > >> believe
> > >> >> > that there are no serious performance or expressivity killers
> when
> > >> >> > comparing alternative serialization frameworks.  As far as I
> know,
> > >> >> > performance differences are not massive (and protobufs is one of
> > the
> > >> >> > leaders in any case) and the expressivity differences are
> > essentially
> > >> >> nil.
> > >> >> >  If somebody feels that there is a serious show-stopper with any
> > >> option,
> > >> >> > they should speak.
> > >> >> >
> > >> >> > The second is to ask the sense of the community whether they
> judge
> > >> >> progress
> > >> >> > or perfection in this decision is most important to the project.
> >  My
> > >> >> guess
> > >> >> > is that almost everybody would prefer to see progress as long as
> > the
> > >> >> > technical choice is not subject to some horrid missing bit.
> > >> >> >
> > >> >> > The final question is whether it is reasonable to go along with
> > >> protobufs
> > >> >> > given that several very experienced engineers prefer it and would
> > >> like to
> > >> >> > produce code based on it.  If the first two answers are answered
> to
> > >> the
> > >> >> > effect of protobufs is about as good as we will find and that
> > progress
> > >> >> > trumps small differences, then it seems that moving to follow
> this
> > >> >> > preference of Jason and Ryan for protobufs might be a reasonable
> > >> thing to
> > >> >> > do.
> > >> >> >
> > >> >> > The question of an internal wire format, btw, does not constrain
> > the
> > >> >> > project relative to external access.  I think it is important to
> > >> support
> > >> >> > JDBC and ODBC and whatever is in common use for querying.  For
> > >> external
> > >> >> > access the question is quite different.  Whereas for the internal
> > >> format
> > >> >> > consensus around a single choice has large benefits, the external
> > >> format
> > >> >> > choice is nearly the opposite.  For an external format, limiting
> > >> >> ourselves
> > >> >> > to a single choice seems like a bad idea and increasing the
> > audience
> > >> >> seems
> > >> >> > like a better choice.
> > >> >> >
> > >> >> > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <
> ryanobjc@gmail.com>
> > >> >> wrote:
> > >> >> >
> > >> >> >> Hi folks,
> > >> >> >>
> > >> >> >> I just commented on this first JIRA.  Here is my text:
> > >> >> >>
> > >> >> >> This issue has been hashed over a lot in the Hadoop projects.
> > There
> > >> >> >> was work done to compare thrift vs avro vs protobuf. The
> > conclusion
> > >> >> >> was protobuf was the decision to use.
> > >> >> >>
> > >> >> >> Prior to this move, there had been a lot of noise about
> pluggable
> > RPC
> > >> >> >> transports, and whatnot. It held up adoption of a backwards
> > >> compatible
> > >> >> >> serialization framework for a long time. The problem ended up
> > being
> > >> >> >> the analysis-paralysis, rather than the specific implementation
> > >> >> >> problem. In other words, the problem was a LACK of
> implementation
> > >> than
> > >> >> >> actual REAL problems.
> > >> >> >>
> > >> >> >> Based on this experience, I'd strongly suggest adopting protobuf
> > and
> > >> >> >> moving on. Forget about pluggable RPC implementations, the
> > complexity
> > >> >> >> doesnt deliver benefits. The benefits of protobuf is that its
> the
> > RPC
> > >> >> >> format for Hadoop and HBase, which allows Drill to draw on the
> > broad
> > >> >> >> experience of those communities who need to implement high
> > >> performance
> > >> >> >> backwards compatible RPC serialization.
> > >> >> >>
> > >> >> >> ====
> > >> >> >>
> > >> >> >> Expanding a bit, I've looked in to this issue a lot, and there
> is
> > >> very
> > >> >> >> few significant concrete reasons to choose protobuf vs thrift.
> >  Tiny
> > >> >> >> percent faster of this, and that, etc.  I'd strongly suggest
> > protobuf
> > >> >> >> for the expanded community.  There is no particular Apache
> > imperative
> > >> >> >> that Apache projects re-use libraries.  Use what makes sense for
> > your
> > >> >> >> project.
> > >> >> >>
> > >> >> >> As regards to Avro, it's a fine serialization format for long
> term
> > >> >> >> data retention, but the complexities that exist to enable that
> > make
> > >> it
> > >> >> >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any
> > form.
> > >> >> >>
> > >> >> >> -ryan
> > >> >> >>
> > >> >> >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> > tshiran@maprtech.com>
> > >> >> >> wrote:
> > >> >> >> > We plan to propose the architecture and interfaces in the next
> > >> couple
> > >> >> >> > weeks, which will make it easy to divide the project into
> clear
> > >> >> building
> > >> >> >> > blocks. At that point it will be easier to start contributing
> > >> >> different
> > >> >> >> > data sources, data formats, operators, query languages, etc.
> > >> >> >> >
> > >> >> >> > The contributions are done in the usual Apache way. It's best
> to
> > >> open
> > >> >> a
> > >> >> >> > JIRA and then post a patch so that others can review and then
> a
> > >> >> committer
> > >> >> >> > can check it in.
> > >> >> >> >
> > >> >> >> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> > >> >> >> chandanmadhesia@gmail.com
> > >> >> >> >> wrote:
> > >> >> >> >
> > >> >> >> >> Hi
> > >> >> >> >>
> > >> >> >> >> Hi
> > >> >> >> >>
> > >> >> >> >> What is the process to become a contributor to drill ?
> > >> >> >> >>
> > >> >> >> >> Regards
> > >> >> >> >> chandan
> > >> >> >> >>
> > >> >> >> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> > >> ted.dunning@gmail.com>
> > >> >> >> wrote:
> > >> >> >> >>
> > >> >> >> >> > Suffice it to say that if *you* think it is important
> enough
> > to
> > >> >> >> implement
> > >> >> >> >> > and maintain, then the group shouldn't say naye.  The
> > consensus
> > >> >> stuff
> > >> >> >> >> > should only block things that break something else.
>  Additive
> > >> >> features
> > >> >> >> >> that
> > >> >> >> >> > are highly maintainable (or which come with commitments)
> > >> shouldn't
> > >> >> >> >> > generally be blocked.
> > >> >> >> >> >
> > >> >> >> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > >> >> >> >> > michael.hausenblas@gmail.com> wrote:
> > >> >> >> >> >
> > >> >> >> >> > > Good. Feel free to put me down for that, if the group as
> a
> > >> whole
> > >> >> >> thinks
> > >> >> >> >> > > that (supporting Thrift) makes sense.
> > >> >> >> >> > >
> > >> >> >> >> >
> > >> >> >> >>
> > >> >> >> >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > --
> > >> >> >> > Tomer Shiran
> > >> >> >> > Director of Product Management | MapR Technologies |
> > 650-804-8657
> > >> >> >>
> > >> >>
> > >>
> >
>

Re: Thrift?

Posted by Constantine Peresypkin <pc...@gmail.com>.
I really have no idea how one can estimate telco traffic.
But I highly doubt that you can fruitfully compare reliability of
internal-only protocol (same implementation, easy to enforce compatibility)
to an interoperable one.

On Sat, Sep 15, 2012 at 1:41 AM, Ryan Rawson <ry...@gmail.com> wrote:

> I didn't say I was the one making the argument...
>
> Google has put probably > 10^24 bytes of data thru protobuf in
> multiple implementations (eg: serialization on disk and on wire RPC).
>   That is a low estimate.
>
> I'd be interested in hearing what 20 years of telco protocol traffic
> might compare to 10 years of google's usage of protobuf.  Exponential
> curve and all of that.
>
>
>
>
>
> On Fri, Sep 14, 2012 at 3:36 PM, Constantine Peresypkin
> <pc...@gmail.com> wrote:
> > More battle tested than more than 20 year old standard used almost in
> every
> > telecom protocol that exists nowdays?
> > I think your statement is a little on "too bold" side. :)
> >
> > On Sat, Sep 15, 2012 at 1:30 AM, Ryan Rawson <ry...@gmail.com> wrote:
> >
> >> Funny thing, given how much use protobufs has been put thru, I think
> >> one could make the argument its more battle tested than ASN.1 ...
> >>
> >> On Fri, Sep 14, 2012 at 3:24 PM, Constantine Peresypkin
> >> <co...@litestack.com> wrote:
> >> > Protobuf is an attempt to make ASN.1 more developer friendly (not a
> bad
> >> > attempt).
> >> > It's simpler, has much less features, easier to implement and has a
> >> compact
> >> > encoding.
> >> > But on other hand it's non-standard, "reinvented wheel" they could
> just
> >> do
> >> > a "better than PER" encoding for ASN.1, and AFAIK has no support for
> the
> >> > new and shiny Google encodings, like "group varint".
> >> > All in all in current situation it seems a better choice than ASN.1,
> not
> >> > even arguing about something even more vague and non-standard as
> Thrift.
> >> >
> >> > On Sat, Sep 15, 2012 at 12:38 AM, Ryan Rawson <ry...@gmail.com>
> >> wrote:
> >> >
> >> >> Thanks for that Ted.
> >> >>
> >> >> Correct - internal wire format doesnt mean 'drill only supports
> >> >> protobuf encoded data'.
> >> >>
> >> >> Part of the reason to favor protobuf is that a lot of people in the
> >> >> broader 'big data' community are building a lot of experience with
> it.
> >> >>  Hadoop and HBase both are moving to/moved to protobuf on the wire.
> >> >> Being able to leverage this expertise is valuable.
> >> >>
> >> >> There is a JIRA in Hadoop-land where someone had done a deep dive
> >> >> 'bake off' between thrift, protobuf and avro.  The ultimate choice
> was
> >> >> protobuf for a number of reasons.  If people want to re-do the
> >> >> analysis, I'd like to see it in the context of THAT analysis (eg: why
> >> >> the assumptions there are not the same for Drill)... if anything it'd
> >> >> give a concrete form to what can be a mire.
> >> >>
> >> >> For what it's worth, I've had many discussion along these angles with
> >> >> a variety of people including committers on Thrift, and the consensus
> >> >> is both are good choices.
> >> >>
> >> >> -ryan
> >> >>
> >> >> On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <te...@gmail.com>
> >> >> wrote:
> >> >> > I think that it is important to ask a few questions leading up a
> >> decision
> >> >> > here.
> >> >> >
> >> >> > The first is a (rhetorical) show of hands about how many people
> >> believe
> >> >> > that there are no serious performance or expressivity killers when
> >> >> > comparing alternative serialization frameworks.  As far as I know,
> >> >> > performance differences are not massive (and protobufs is one of
> the
> >> >> > leaders in any case) and the expressivity differences are
> essentially
> >> >> nil.
> >> >> >  If somebody feels that there is a serious show-stopper with any
> >> option,
> >> >> > they should speak.
> >> >> >
> >> >> > The second is to ask the sense of the community whether they judge
> >> >> progress
> >> >> > or perfection in this decision is most important to the project.
>  My
> >> >> guess
> >> >> > is that almost everybody would prefer to see progress as long as
> the
> >> >> > technical choice is not subject to some horrid missing bit.
> >> >> >
> >> >> > The final question is whether it is reasonable to go along with
> >> protobufs
> >> >> > given that several very experienced engineers prefer it and would
> >> like to
> >> >> > produce code based on it.  If the first two answers are answered to
> >> the
> >> >> > effect of protobufs is about as good as we will find and that
> progress
> >> >> > trumps small differences, then it seems that moving to follow this
> >> >> > preference of Jason and Ryan for protobufs might be a reasonable
> >> thing to
> >> >> > do.
> >> >> >
> >> >> > The question of an internal wire format, btw, does not constrain
> the
> >> >> > project relative to external access.  I think it is important to
> >> support
> >> >> > JDBC and ODBC and whatever is in common use for querying.  For
> >> external
> >> >> > access the question is quite different.  Whereas for the internal
> >> format
> >> >> > consensus around a single choice has large benefits, the external
> >> format
> >> >> > choice is nearly the opposite.  For an external format, limiting
> >> >> ourselves
> >> >> > to a single choice seems like a bad idea and increasing the
> audience
> >> >> seems
> >> >> > like a better choice.
> >> >> >
> >> >> > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Hi folks,
> >> >> >>
> >> >> >> I just commented on this first JIRA.  Here is my text:
> >> >> >>
> >> >> >> This issue has been hashed over a lot in the Hadoop projects.
> There
> >> >> >> was work done to compare thrift vs avro vs protobuf. The
> conclusion
> >> >> >> was protobuf was the decision to use.
> >> >> >>
> >> >> >> Prior to this move, there had been a lot of noise about pluggable
> RPC
> >> >> >> transports, and whatnot. It held up adoption of a backwards
> >> compatible
> >> >> >> serialization framework for a long time. The problem ended up
> being
> >> >> >> the analysis-paralysis, rather than the specific implementation
> >> >> >> problem. In other words, the problem was a LACK of implementation
> >> than
> >> >> >> actual REAL problems.
> >> >> >>
> >> >> >> Based on this experience, I'd strongly suggest adopting protobuf
> and
> >> >> >> moving on. Forget about pluggable RPC implementations, the
> complexity
> >> >> >> doesnt deliver benefits. The benefits of protobuf is that its the
> RPC
> >> >> >> format for Hadoop and HBase, which allows Drill to draw on the
> broad
> >> >> >> experience of those communities who need to implement high
> >> performance
> >> >> >> backwards compatible RPC serialization.
> >> >> >>
> >> >> >> ====
> >> >> >>
> >> >> >> Expanding a bit, I've looked in to this issue a lot, and there is
> >> very
> >> >> >> few significant concrete reasons to choose protobuf vs thrift.
>  Tiny
> >> >> >> percent faster of this, and that, etc.  I'd strongly suggest
> protobuf
> >> >> >> for the expanded community.  There is no particular Apache
> imperative
> >> >> >> that Apache projects re-use libraries.  Use what makes sense for
> your
> >> >> >> project.
> >> >> >>
> >> >> >> As regards to Avro, it's a fine serialization format for long term
> >> >> >> data retention, but the complexities that exist to enable that
> make
> >> it
> >> >> >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any
> form.
> >> >> >>
> >> >> >> -ryan
> >> >> >>
> >> >> >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <
> tshiran@maprtech.com>
> >> >> >> wrote:
> >> >> >> > We plan to propose the architecture and interfaces in the next
> >> couple
> >> >> >> > weeks, which will make it easy to divide the project into clear
> >> >> building
> >> >> >> > blocks. At that point it will be easier to start contributing
> >> >> different
> >> >> >> > data sources, data formats, operators, query languages, etc.
> >> >> >> >
> >> >> >> > The contributions are done in the usual Apache way. It's best to
> >> open
> >> >> a
> >> >> >> > JIRA and then post a patch so that others can review and then a
> >> >> committer
> >> >> >> > can check it in.
> >> >> >> >
> >> >> >> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> >> >> >> chandanmadhesia@gmail.com
> >> >> >> >> wrote:
> >> >> >> >
> >> >> >> >> Hi
> >> >> >> >>
> >> >> >> >> Hi
> >> >> >> >>
> >> >> >> >> What is the process to become a contributor to drill ?
> >> >> >> >>
> >> >> >> >> Regards
> >> >> >> >> chandan
> >> >> >> >>
> >> >> >> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> >> ted.dunning@gmail.com>
> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >> > Suffice it to say that if *you* think it is important enough
> to
> >> >> >> implement
> >> >> >> >> > and maintain, then the group shouldn't say naye.  The
> consensus
> >> >> stuff
> >> >> >> >> > should only block things that break something else.  Additive
> >> >> features
> >> >> >> >> that
> >> >> >> >> > are highly maintainable (or which come with commitments)
> >> shouldn't
> >> >> >> >> > generally be blocked.
> >> >> >> >> >
> >> >> >> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> >> >> >> >> > michael.hausenblas@gmail.com> wrote:
> >> >> >> >> >
> >> >> >> >> > > Good. Feel free to put me down for that, if the group as a
> >> whole
> >> >> >> thinks
> >> >> >> >> > > that (supporting Thrift) makes sense.
> >> >> >> >> > >
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > Tomer Shiran
> >> >> >> > Director of Product Management | MapR Technologies |
> 650-804-8657
> >> >> >>
> >> >>
> >>
>

Re: Thrift?

Posted by Ryan Rawson <ry...@gmail.com>.
I didn't say I was the one making the argument...

Google has put probably > 10^24 bytes of data thru protobuf in
multiple implementations (eg: serialization on disk and on wire RPC).
  That is a low estimate.

I'd be interested in hearing what 20 years of telco protocol traffic
might compare to 10 years of google's usage of protobuf.  Exponential
curve and all of that.





On Fri, Sep 14, 2012 at 3:36 PM, Constantine Peresypkin
<pc...@gmail.com> wrote:
> More battle tested than more than 20 year old standard used almost in every
> telecom protocol that exists nowdays?
> I think your statement is a little on "too bold" side. :)
>
> On Sat, Sep 15, 2012 at 1:30 AM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Funny thing, given how much use protobufs has been put thru, I think
>> one could make the argument its more battle tested than ASN.1 ...
>>
>> On Fri, Sep 14, 2012 at 3:24 PM, Constantine Peresypkin
>> <co...@litestack.com> wrote:
>> > Protobuf is an attempt to make ASN.1 more developer friendly (not a bad
>> > attempt).
>> > It's simpler, has much less features, easier to implement and has a
>> compact
>> > encoding.
>> > But on other hand it's non-standard, "reinvented wheel" they could just
>> do
>> > a "better than PER" encoding for ASN.1, and AFAIK has no support for the
>> > new and shiny Google encodings, like "group varint".
>> > All in all in current situation it seems a better choice than ASN.1, not
>> > even arguing about something even more vague and non-standard as Thrift.
>> >
>> > On Sat, Sep 15, 2012 at 12:38 AM, Ryan Rawson <ry...@gmail.com>
>> wrote:
>> >
>> >> Thanks for that Ted.
>> >>
>> >> Correct - internal wire format doesnt mean 'drill only supports
>> >> protobuf encoded data'.
>> >>
>> >> Part of the reason to favor protobuf is that a lot of people in the
>> >> broader 'big data' community are building a lot of experience with it.
>> >>  Hadoop and HBase both are moving to/moved to protobuf on the wire.
>> >> Being able to leverage this expertise is valuable.
>> >>
>> >> There is a JIRA in Hadoop-land where someone had done a deep dive
>> >> 'bake off' between thrift, protobuf and avro.  The ultimate choice was
>> >> protobuf for a number of reasons.  If people want to re-do the
>> >> analysis, I'd like to see it in the context of THAT analysis (eg: why
>> >> the assumptions there are not the same for Drill)... if anything it'd
>> >> give a concrete form to what can be a mire.
>> >>
>> >> For what it's worth, I've had many discussion along these angles with
>> >> a variety of people including committers on Thrift, and the consensus
>> >> is both are good choices.
>> >>
>> >> -ryan
>> >>
>> >> On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <te...@gmail.com>
>> >> wrote:
>> >> > I think that it is important to ask a few questions leading up a
>> decision
>> >> > here.
>> >> >
>> >> > The first is a (rhetorical) show of hands about how many people
>> believe
>> >> > that there are no serious performance or expressivity killers when
>> >> > comparing alternative serialization frameworks.  As far as I know,
>> >> > performance differences are not massive (and protobufs is one of the
>> >> > leaders in any case) and the expressivity differences are essentially
>> >> nil.
>> >> >  If somebody feels that there is a serious show-stopper with any
>> option,
>> >> > they should speak.
>> >> >
>> >> > The second is to ask the sense of the community whether they judge
>> >> progress
>> >> > or perfection in this decision is most important to the project.  My
>> >> guess
>> >> > is that almost everybody would prefer to see progress as long as the
>> >> > technical choice is not subject to some horrid missing bit.
>> >> >
>> >> > The final question is whether it is reasonable to go along with
>> protobufs
>> >> > given that several very experienced engineers prefer it and would
>> like to
>> >> > produce code based on it.  If the first two answers are answered to
>> the
>> >> > effect of protobufs is about as good as we will find and that progress
>> >> > trumps small differences, then it seems that moving to follow this
>> >> > preference of Jason and Ryan for protobufs might be a reasonable
>> thing to
>> >> > do.
>> >> >
>> >> > The question of an internal wire format, btw, does not constrain the
>> >> > project relative to external access.  I think it is important to
>> support
>> >> > JDBC and ODBC and whatever is in common use for querying.  For
>> external
>> >> > access the question is quite different.  Whereas for the internal
>> format
>> >> > consensus around a single choice has large benefits, the external
>> format
>> >> > choice is nearly the opposite.  For an external format, limiting
>> >> ourselves
>> >> > to a single choice seems like a bad idea and increasing the audience
>> >> seems
>> >> > like a better choice.
>> >> >
>> >> > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Hi folks,
>> >> >>
>> >> >> I just commented on this first JIRA.  Here is my text:
>> >> >>
>> >> >> This issue has been hashed over a lot in the Hadoop projects. There
>> >> >> was work done to compare thrift vs avro vs protobuf. The conclusion
>> >> >> was protobuf was the decision to use.
>> >> >>
>> >> >> Prior to this move, there had been a lot of noise about pluggable RPC
>> >> >> transports, and whatnot. It held up adoption of a backwards
>> compatible
>> >> >> serialization framework for a long time. The problem ended up being
>> >> >> the analysis-paralysis, rather than the specific implementation
>> >> >> problem. In other words, the problem was a LACK of implementation
>> than
>> >> >> actual REAL problems.
>> >> >>
>> >> >> Based on this experience, I'd strongly suggest adopting protobuf and
>> >> >> moving on. Forget about pluggable RPC implementations, the complexity
>> >> >> doesnt deliver benefits. The benefits of protobuf is that its the RPC
>> >> >> format for Hadoop and HBase, which allows Drill to draw on the broad
>> >> >> experience of those communities who need to implement high
>> performance
>> >> >> backwards compatible RPC serialization.
>> >> >>
>> >> >> ====
>> >> >>
>> >> >> Expanding a bit, I've looked in to this issue a lot, and there is
>> very
>> >> >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
>> >> >> percent faster of this, and that, etc.  I'd strongly suggest protobuf
>> >> >> for the expanded community.  There is no particular Apache imperative
>> >> >> that Apache projects re-use libraries.  Use what makes sense for your
>> >> >> project.
>> >> >>
>> >> >> As regards to Avro, it's a fine serialization format for long term
>> >> >> data retention, but the complexities that exist to enable that make
>> it
>> >> >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
>> >> >>
>> >> >> -ryan
>> >> >>
>> >> >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
>> >> >> wrote:
>> >> >> > We plan to propose the architecture and interfaces in the next
>> couple
>> >> >> > weeks, which will make it easy to divide the project into clear
>> >> building
>> >> >> > blocks. At that point it will be easier to start contributing
>> >> different
>> >> >> > data sources, data formats, operators, query languages, etc.
>> >> >> >
>> >> >> > The contributions are done in the usual Apache way. It's best to
>> open
>> >> a
>> >> >> > JIRA and then post a patch so that others can review and then a
>> >> committer
>> >> >> > can check it in.
>> >> >> >
>> >> >> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
>> >> >> chandanmadhesia@gmail.com
>> >> >> >> wrote:
>> >> >> >
>> >> >> >> Hi
>> >> >> >>
>> >> >> >> Hi
>> >> >> >>
>> >> >> >> What is the process to become a contributor to drill ?
>> >> >> >>
>> >> >> >> Regards
>> >> >> >> chandan
>> >> >> >>
>> >> >> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
>> ted.dunning@gmail.com>
>> >> >> wrote:
>> >> >> >>
>> >> >> >> > Suffice it to say that if *you* think it is important enough to
>> >> >> implement
>> >> >> >> > and maintain, then the group shouldn't say naye.  The consensus
>> >> stuff
>> >> >> >> > should only block things that break something else.  Additive
>> >> features
>> >> >> >> that
>> >> >> >> > are highly maintainable (or which come with commitments)
>> shouldn't
>> >> >> >> > generally be blocked.
>> >> >> >> >
>> >> >> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
>> >> >> >> > michael.hausenblas@gmail.com> wrote:
>> >> >> >> >
>> >> >> >> > > Good. Feel free to put me down for that, if the group as a
>> whole
>> >> >> thinks
>> >> >> >> > > that (supporting Thrift) makes sense.
>> >> >> >> > >
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Tomer Shiran
>> >> >> > Director of Product Management | MapR Technologies | 650-804-8657
>> >> >>
>> >>
>>

Re: Thrift?

Posted by Constantine Peresypkin <pc...@gmail.com>.
More battle tested than more than 20 year old standard used almost in every
telecom protocol that exists nowdays?
I think your statement is a little on "too bold" side. :)

On Sat, Sep 15, 2012 at 1:30 AM, Ryan Rawson <ry...@gmail.com> wrote:

> Funny thing, given how much use protobufs has been put thru, I think
> one could make the argument its more battle tested than ASN.1 ...
>
> On Fri, Sep 14, 2012 at 3:24 PM, Constantine Peresypkin
> <co...@litestack.com> wrote:
> > Protobuf is an attempt to make ASN.1 more developer friendly (not a bad
> > attempt).
> > It's simpler, has much less features, easier to implement and has a
> compact
> > encoding.
> > But on other hand it's non-standard, "reinvented wheel" they could just
> do
> > a "better than PER" encoding for ASN.1, and AFAIK has no support for the
> > new and shiny Google encodings, like "group varint".
> > All in all in current situation it seems a better choice than ASN.1, not
> > even arguing about something even more vague and non-standard as Thrift.
> >
> > On Sat, Sep 15, 2012 at 12:38 AM, Ryan Rawson <ry...@gmail.com>
> wrote:
> >
> >> Thanks for that Ted.
> >>
> >> Correct - internal wire format doesnt mean 'drill only supports
> >> protobuf encoded data'.
> >>
> >> Part of the reason to favor protobuf is that a lot of people in the
> >> broader 'big data' community are building a lot of experience with it.
> >>  Hadoop and HBase both are moving to/moved to protobuf on the wire.
> >> Being able to leverage this expertise is valuable.
> >>
> >> There is a JIRA in Hadoop-land where someone had done a deep dive
> >> 'bake off' between thrift, protobuf and avro.  The ultimate choice was
> >> protobuf for a number of reasons.  If people want to re-do the
> >> analysis, I'd like to see it in the context of THAT analysis (eg: why
> >> the assumptions there are not the same for Drill)... if anything it'd
> >> give a concrete form to what can be a mire.
> >>
> >> For what it's worth, I've had many discussion along these angles with
> >> a variety of people including committers on Thrift, and the consensus
> >> is both are good choices.
> >>
> >> -ryan
> >>
> >> On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <te...@gmail.com>
> >> wrote:
> >> > I think that it is important to ask a few questions leading up a
> decision
> >> > here.
> >> >
> >> > The first is a (rhetorical) show of hands about how many people
> believe
> >> > that there are no serious performance or expressivity killers when
> >> > comparing alternative serialization frameworks.  As far as I know,
> >> > performance differences are not massive (and protobufs is one of the
> >> > leaders in any case) and the expressivity differences are essentially
> >> nil.
> >> >  If somebody feels that there is a serious show-stopper with any
> option,
> >> > they should speak.
> >> >
> >> > The second is to ask the sense of the community whether they judge
> >> progress
> >> > or perfection in this decision is most important to the project.  My
> >> guess
> >> > is that almost everybody would prefer to see progress as long as the
> >> > technical choice is not subject to some horrid missing bit.
> >> >
> >> > The final question is whether it is reasonable to go along with
> protobufs
> >> > given that several very experienced engineers prefer it and would
> like to
> >> > produce code based on it.  If the first two answers are answered to
> the
> >> > effect of protobufs is about as good as we will find and that progress
> >> > trumps small differences, then it seems that moving to follow this
> >> > preference of Jason and Ryan for protobufs might be a reasonable
> thing to
> >> > do.
> >> >
> >> > The question of an internal wire format, btw, does not constrain the
> >> > project relative to external access.  I think it is important to
> support
> >> > JDBC and ODBC and whatever is in common use for querying.  For
> external
> >> > access the question is quite different.  Whereas for the internal
> format
> >> > consensus around a single choice has large benefits, the external
> format
> >> > choice is nearly the opposite.  For an external format, limiting
> >> ourselves
> >> > to a single choice seems like a bad idea and increasing the audience
> >> seems
> >> > like a better choice.
> >> >
> >> > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> >> wrote:
> >> >
> >> >> Hi folks,
> >> >>
> >> >> I just commented on this first JIRA.  Here is my text:
> >> >>
> >> >> This issue has been hashed over a lot in the Hadoop projects. There
> >> >> was work done to compare thrift vs avro vs protobuf. The conclusion
> >> >> was protobuf was the decision to use.
> >> >>
> >> >> Prior to this move, there had been a lot of noise about pluggable RPC
> >> >> transports, and whatnot. It held up adoption of a backwards
> compatible
> >> >> serialization framework for a long time. The problem ended up being
> >> >> the analysis-paralysis, rather than the specific implementation
> >> >> problem. In other words, the problem was a LACK of implementation
> than
> >> >> actual REAL problems.
> >> >>
> >> >> Based on this experience, I'd strongly suggest adopting protobuf and
> >> >> moving on. Forget about pluggable RPC implementations, the complexity
> >> >> doesnt deliver benefits. The benefits of protobuf is that its the RPC
> >> >> format for Hadoop and HBase, which allows Drill to draw on the broad
> >> >> experience of those communities who need to implement high
> performance
> >> >> backwards compatible RPC serialization.
> >> >>
> >> >> ====
> >> >>
> >> >> Expanding a bit, I've looked in to this issue a lot, and there is
> very
> >> >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> >> >> percent faster of this, and that, etc.  I'd strongly suggest protobuf
> >> >> for the expanded community.  There is no particular Apache imperative
> >> >> that Apache projects re-use libraries.  Use what makes sense for your
> >> >> project.
> >> >>
> >> >> As regards to Avro, it's a fine serialization format for long term
> >> >> data retention, but the complexities that exist to enable that make
> it
> >> >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
> >> >>
> >> >> -ryan
> >> >>
> >> >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
> >> >> wrote:
> >> >> > We plan to propose the architecture and interfaces in the next
> couple
> >> >> > weeks, which will make it easy to divide the project into clear
> >> building
> >> >> > blocks. At that point it will be easier to start contributing
> >> different
> >> >> > data sources, data formats, operators, query languages, etc.
> >> >> >
> >> >> > The contributions are done in the usual Apache way. It's best to
> open
> >> a
> >> >> > JIRA and then post a patch so that others can review and then a
> >> committer
> >> >> > can check it in.
> >> >> >
> >> >> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> >> >> chandanmadhesia@gmail.com
> >> >> >> wrote:
> >> >> >
> >> >> >> Hi
> >> >> >>
> >> >> >> Hi
> >> >> >>
> >> >> >> What is the process to become a contributor to drill ?
> >> >> >>
> >> >> >> Regards
> >> >> >> chandan
> >> >> >>
> >> >> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <
> ted.dunning@gmail.com>
> >> >> wrote:
> >> >> >>
> >> >> >> > Suffice it to say that if *you* think it is important enough to
> >> >> implement
> >> >> >> > and maintain, then the group shouldn't say naye.  The consensus
> >> stuff
> >> >> >> > should only block things that break something else.  Additive
> >> features
> >> >> >> that
> >> >> >> > are highly maintainable (or which come with commitments)
> shouldn't
> >> >> >> > generally be blocked.
> >> >> >> >
> >> >> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> >> >> >> > michael.hausenblas@gmail.com> wrote:
> >> >> >> >
> >> >> >> > > Good. Feel free to put me down for that, if the group as a
> whole
> >> >> thinks
> >> >> >> > > that (supporting Thrift) makes sense.
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Tomer Shiran
> >> >> > Director of Product Management | MapR Technologies | 650-804-8657
> >> >>
> >>
>

Re: Thrift?

Posted by Ryan Rawson <ry...@gmail.com>.
Funny thing, given how much use protobufs has been put thru, I think
one could make the argument its more battle tested than ASN.1 ...

On Fri, Sep 14, 2012 at 3:24 PM, Constantine Peresypkin
<co...@litestack.com> wrote:
> Protobuf is an attempt to make ASN.1 more developer friendly (not a bad
> attempt).
> It's simpler, has much less features, easier to implement and has a compact
> encoding.
> But on other hand it's non-standard, "reinvented wheel" they could just do
> a "better than PER" encoding for ASN.1, and AFAIK has no support for the
> new and shiny Google encodings, like "group varint".
> All in all in current situation it seems a better choice than ASN.1, not
> even arguing about something even more vague and non-standard as Thrift.
>
> On Sat, Sep 15, 2012 at 12:38 AM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Thanks for that Ted.
>>
>> Correct - internal wire format doesnt mean 'drill only supports
>> protobuf encoded data'.
>>
>> Part of the reason to favor protobuf is that a lot of people in the
>> broader 'big data' community are building a lot of experience with it.
>>  Hadoop and HBase both are moving to/moved to protobuf on the wire.
>> Being able to leverage this expertise is valuable.
>>
>> There is a JIRA in Hadoop-land where someone had done a deep dive
>> 'bake off' between thrift, protobuf and avro.  The ultimate choice was
>> protobuf for a number of reasons.  If people want to re-do the
>> analysis, I'd like to see it in the context of THAT analysis (eg: why
>> the assumptions there are not the same for Drill)... if anything it'd
>> give a concrete form to what can be a mire.
>>
>> For what it's worth, I've had many discussion along these angles with
>> a variety of people including committers on Thrift, and the consensus
>> is both are good choices.
>>
>> -ryan
>>
>> On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>> > I think that it is important to ask a few questions leading up a decision
>> > here.
>> >
>> > The first is a (rhetorical) show of hands about how many people believe
>> > that there are no serious performance or expressivity killers when
>> > comparing alternative serialization frameworks.  As far as I know,
>> > performance differences are not massive (and protobufs is one of the
>> > leaders in any case) and the expressivity differences are essentially
>> nil.
>> >  If somebody feels that there is a serious show-stopper with any option,
>> > they should speak.
>> >
>> > The second is to ask the sense of the community whether they judge
>> progress
>> > or perfection in this decision is most important to the project.  My
>> guess
>> > is that almost everybody would prefer to see progress as long as the
>> > technical choice is not subject to some horrid missing bit.
>> >
>> > The final question is whether it is reasonable to go along with protobufs
>> > given that several very experienced engineers prefer it and would like to
>> > produce code based on it.  If the first two answers are answered to the
>> > effect of protobufs is about as good as we will find and that progress
>> > trumps small differences, then it seems that moving to follow this
>> > preference of Jason and Ryan for protobufs might be a reasonable thing to
>> > do.
>> >
>> > The question of an internal wire format, btw, does not constrain the
>> > project relative to external access.  I think it is important to support
>> > JDBC and ODBC and whatever is in common use for querying.  For external
>> > access the question is quite different.  Whereas for the internal format
>> > consensus around a single choice has large benefits, the external format
>> > choice is nearly the opposite.  For an external format, limiting
>> ourselves
>> > to a single choice seems like a bad idea and increasing the audience
>> seems
>> > like a better choice.
>> >
>> > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
>> wrote:
>> >
>> >> Hi folks,
>> >>
>> >> I just commented on this first JIRA.  Here is my text:
>> >>
>> >> This issue has been hashed over a lot in the Hadoop projects. There
>> >> was work done to compare thrift vs avro vs protobuf. The conclusion
>> >> was protobuf was the decision to use.
>> >>
>> >> Prior to this move, there had been a lot of noise about pluggable RPC
>> >> transports, and whatnot. It held up adoption of a backwards compatible
>> >> serialization framework for a long time. The problem ended up being
>> >> the analysis-paralysis, rather than the specific implementation
>> >> problem. In other words, the problem was a LACK of implementation than
>> >> actual REAL problems.
>> >>
>> >> Based on this experience, I'd strongly suggest adopting protobuf and
>> >> moving on. Forget about pluggable RPC implementations, the complexity
>> >> doesnt deliver benefits. The benefits of protobuf is that its the RPC
>> >> format for Hadoop and HBase, which allows Drill to draw on the broad
>> >> experience of those communities who need to implement high performance
>> >> backwards compatible RPC serialization.
>> >>
>> >> ====
>> >>
>> >> Expanding a bit, I've looked in to this issue a lot, and there is very
>> >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
>> >> percent faster of this, and that, etc.  I'd strongly suggest protobuf
>> >> for the expanded community.  There is no particular Apache imperative
>> >> that Apache projects re-use libraries.  Use what makes sense for your
>> >> project.
>> >>
>> >> As regards to Avro, it's a fine serialization format for long term
>> >> data retention, but the complexities that exist to enable that make it
>> >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
>> >>
>> >> -ryan
>> >>
>> >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
>> >> wrote:
>> >> > We plan to propose the architecture and interfaces in the next couple
>> >> > weeks, which will make it easy to divide the project into clear
>> building
>> >> > blocks. At that point it will be easier to start contributing
>> different
>> >> > data sources, data formats, operators, query languages, etc.
>> >> >
>> >> > The contributions are done in the usual Apache way. It's best to open
>> a
>> >> > JIRA and then post a patch so that others can review and then a
>> committer
>> >> > can check it in.
>> >> >
>> >> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
>> >> chandanmadhesia@gmail.com
>> >> >> wrote:
>> >> >
>> >> >> Hi
>> >> >>
>> >> >> Hi
>> >> >>
>> >> >> What is the process to become a contributor to drill ?
>> >> >>
>> >> >> Regards
>> >> >> chandan
>> >> >>
>> >> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
>> >> wrote:
>> >> >>
>> >> >> > Suffice it to say that if *you* think it is important enough to
>> >> implement
>> >> >> > and maintain, then the group shouldn't say naye.  The consensus
>> stuff
>> >> >> > should only block things that break something else.  Additive
>> features
>> >> >> that
>> >> >> > are highly maintainable (or which come with commitments) shouldn't
>> >> >> > generally be blocked.
>> >> >> >
>> >> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
>> >> >> > michael.hausenblas@gmail.com> wrote:
>> >> >> >
>> >> >> > > Good. Feel free to put me down for that, if the group as a whole
>> >> thinks
>> >> >> > > that (supporting Thrift) makes sense.
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Tomer Shiran
>> >> > Director of Product Management | MapR Technologies | 650-804-8657
>> >>
>>

Re: Thrift?

Posted by Constantine Peresypkin <co...@litestack.com>.
Protobuf is an attempt to make ASN.1 more developer friendly (not a bad
attempt).
It's simpler, has much less features, easier to implement and has a compact
encoding.
But on other hand it's non-standard, "reinvented wheel" they could just do
a "better than PER" encoding for ASN.1, and AFAIK has no support for the
new and shiny Google encodings, like "group varint".
All in all in current situation it seems a better choice than ASN.1, not
even arguing about something even more vague and non-standard as Thrift.

On Sat, Sep 15, 2012 at 12:38 AM, Ryan Rawson <ry...@gmail.com> wrote:

> Thanks for that Ted.
>
> Correct - internal wire format doesnt mean 'drill only supports
> protobuf encoded data'.
>
> Part of the reason to favor protobuf is that a lot of people in the
> broader 'big data' community are building a lot of experience with it.
>  Hadoop and HBase both are moving to/moved to protobuf on the wire.
> Being able to leverage this expertise is valuable.
>
> There is a JIRA in Hadoop-land where someone had done a deep dive
> 'bake off' between thrift, protobuf and avro.  The ultimate choice was
> protobuf for a number of reasons.  If people want to re-do the
> analysis, I'd like to see it in the context of THAT analysis (eg: why
> the assumptions there are not the same for Drill)... if anything it'd
> give a concrete form to what can be a mire.
>
> For what it's worth, I've had many discussion along these angles with
> a variety of people including committers on Thrift, and the consensus
> is both are good choices.
>
> -ryan
>
> On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <te...@gmail.com>
> wrote:
> > I think that it is important to ask a few questions leading up a decision
> > here.
> >
> > The first is a (rhetorical) show of hands about how many people believe
> > that there are no serious performance or expressivity killers when
> > comparing alternative serialization frameworks.  As far as I know,
> > performance differences are not massive (and protobufs is one of the
> > leaders in any case) and the expressivity differences are essentially
> nil.
> >  If somebody feels that there is a serious show-stopper with any option,
> > they should speak.
> >
> > The second is to ask the sense of the community whether they judge
> progress
> > or perfection in this decision is most important to the project.  My
> guess
> > is that almost everybody would prefer to see progress as long as the
> > technical choice is not subject to some horrid missing bit.
> >
> > The final question is whether it is reasonable to go along with protobufs
> > given that several very experienced engineers prefer it and would like to
> > produce code based on it.  If the first two answers are answered to the
> > effect of protobufs is about as good as we will find and that progress
> > trumps small differences, then it seems that moving to follow this
> > preference of Jason and Ryan for protobufs might be a reasonable thing to
> > do.
> >
> > The question of an internal wire format, btw, does not constrain the
> > project relative to external access.  I think it is important to support
> > JDBC and ODBC and whatever is in common use for querying.  For external
> > access the question is quite different.  Whereas for the internal format
> > consensus around a single choice has large benefits, the external format
> > choice is nearly the opposite.  For an external format, limiting
> ourselves
> > to a single choice seems like a bad idea and increasing the audience
> seems
> > like a better choice.
> >
> > On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com>
> wrote:
> >
> >> Hi folks,
> >>
> >> I just commented on this first JIRA.  Here is my text:
> >>
> >> This issue has been hashed over a lot in the Hadoop projects. There
> >> was work done to compare thrift vs avro vs protobuf. The conclusion
> >> was protobuf was the decision to use.
> >>
> >> Prior to this move, there had been a lot of noise about pluggable RPC
> >> transports, and whatnot. It held up adoption of a backwards compatible
> >> serialization framework for a long time. The problem ended up being
> >> the analysis-paralysis, rather than the specific implementation
> >> problem. In other words, the problem was a LACK of implementation than
> >> actual REAL problems.
> >>
> >> Based on this experience, I'd strongly suggest adopting protobuf and
> >> moving on. Forget about pluggable RPC implementations, the complexity
> >> doesnt deliver benefits. The benefits of protobuf is that its the RPC
> >> format for Hadoop and HBase, which allows Drill to draw on the broad
> >> experience of those communities who need to implement high performance
> >> backwards compatible RPC serialization.
> >>
> >> ====
> >>
> >> Expanding a bit, I've looked in to this issue a lot, and there is very
> >> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> >> percent faster of this, and that, etc.  I'd strongly suggest protobuf
> >> for the expanded community.  There is no particular Apache imperative
> >> that Apache projects re-use libraries.  Use what makes sense for your
> >> project.
> >>
> >> As regards to Avro, it's a fine serialization format for long term
> >> data retention, but the complexities that exist to enable that make it
> >> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
> >>
> >> -ryan
> >>
> >> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
> >> wrote:
> >> > We plan to propose the architecture and interfaces in the next couple
> >> > weeks, which will make it easy to divide the project into clear
> building
> >> > blocks. At that point it will be easier to start contributing
> different
> >> > data sources, data formats, operators, query languages, etc.
> >> >
> >> > The contributions are done in the usual Apache way. It's best to open
> a
> >> > JIRA and then post a patch so that others can review and then a
> committer
> >> > can check it in.
> >> >
> >> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> >> chandanmadhesia@gmail.com
> >> >> wrote:
> >> >
> >> >> Hi
> >> >>
> >> >> Hi
> >> >>
> >> >> What is the process to become a contributor to drill ?
> >> >>
> >> >> Regards
> >> >> chandan
> >> >>
> >> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
> >> wrote:
> >> >>
> >> >> > Suffice it to say that if *you* think it is important enough to
> >> implement
> >> >> > and maintain, then the group shouldn't say naye.  The consensus
> stuff
> >> >> > should only block things that break something else.  Additive
> features
> >> >> that
> >> >> > are highly maintainable (or which come with commitments) shouldn't
> >> >> > generally be blocked.
> >> >> >
> >> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> >> >> > michael.hausenblas@gmail.com> wrote:
> >> >> >
> >> >> > > Good. Feel free to put me down for that, if the group as a whole
> >> thinks
> >> >> > > that (supporting Thrift) makes sense.
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Tomer Shiran
> >> > Director of Product Management | MapR Technologies | 650-804-8657
> >>
>

Re: Thrift?

Posted by Puneet Zaroo <pu...@gmail.com>.
>
> There is a JIRA in Hadoop-land where someone had done a deep dive
> 'bake off' between thrift, protobuf and avro.  The ultimate choice was
> protobuf for a number of reasons.  If people want to re-do the
> analysis, I'd like to see it in the context of THAT analysis (eg: why
> the assumptions there are not the same for Drill)... if anything it'd
> give a concrete form to what can be a mire.

Could you please provide a pointer to that JIRA. It will be useful
background information for me and perhaps others in the group as well.
thanks,


> On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <te...@gmail.com> wrote:
>> I think that it is important to ask a few questions leading up a decision
>> here.
>>
>> The first is a (rhetorical) show of hands about how many people believe
>> that there are no serious performance or expressivity killers when
>> comparing alternative serialization frameworks.  As far as I know,
>> performance differences are not massive (and protobufs is one of the
>> leaders in any case) and the expressivity differences are essentially nil.
>>  If somebody feels that there is a serious show-stopper with any option,
>> they should speak.
>>
>> The second is to ask the sense of the community whether they judge progress
>> or perfection in this decision is most important to the project.  My guess
>> is that almost everybody would prefer to see progress as long as the
>> technical choice is not subject to some horrid missing bit.
>>
>> The final question is whether it is reasonable to go along with protobufs
>> given that several very experienced engineers prefer it and would like to
>> produce code based on it.  If the first two answers are answered to the
>> effect of protobufs is about as good as we will find and that progress
>> trumps small differences, then it seems that moving to follow this
>> preference of Jason and Ryan for protobufs might be a reasonable thing to
>> do.
>>
>> The question of an internal wire format, btw, does not constrain the
>> project relative to external access.  I think it is important to support
>> JDBC and ODBC and whatever is in common use for querying.  For external
>> access the question is quite different.  Whereas for the internal format
>> consensus around a single choice has large benefits, the external format
>> choice is nearly the opposite.  For an external format, limiting ourselves
>> to a single choice seems like a bad idea and increasing the audience seems
>> like a better choice.
>>
>> On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> I just commented on this first JIRA.  Here is my text:
>>>
>>> This issue has been hashed over a lot in the Hadoop projects. There
>>> was work done to compare thrift vs avro vs protobuf. The conclusion
>>> was protobuf was the decision to use.
>>>
>>> Prior to this move, there had been a lot of noise about pluggable RPC
>>> transports, and whatnot. It held up adoption of a backwards compatible
>>> serialization framework for a long time. The problem ended up being
>>> the analysis-paralysis, rather than the specific implementation
>>> problem. In other words, the problem was a LACK of implementation than
>>> actual REAL problems.
>>>
>>> Based on this experience, I'd strongly suggest adopting protobuf and
>>> moving on. Forget about pluggable RPC implementations, the complexity
>>> doesnt deliver benefits. The benefits of protobuf is that its the RPC
>>> format for Hadoop and HBase, which allows Drill to draw on the broad
>>> experience of those communities who need to implement high performance
>>> backwards compatible RPC serialization.
>>>
>>> ====
>>>
>>> Expanding a bit, I've looked in to this issue a lot, and there is very
>>> few significant concrete reasons to choose protobuf vs thrift.  Tiny
>>> percent faster of this, and that, etc.  I'd strongly suggest protobuf
>>> for the expanded community.  There is no particular Apache imperative
>>> that Apache projects re-use libraries.  Use what makes sense for your
>>> project.
>>>
>>> As regards to Avro, it's a fine serialization format for long term
>>> data retention, but the complexities that exist to enable that make it
>>> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
>>>
>>> -ryan
>>>
>>> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
>>> wrote:
>>> > We plan to propose the architecture and interfaces in the next couple
>>> > weeks, which will make it easy to divide the project into clear building
>>> > blocks. At that point it will be easier to start contributing different
>>> > data sources, data formats, operators, query languages, etc.
>>> >
>>> > The contributions are done in the usual Apache way. It's best to open a
>>> > JIRA and then post a patch so that others can review and then a committer
>>> > can check it in.
>>> >
>>> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
>>> chandanmadhesia@gmail.com
>>> >> wrote:
>>> >
>>> >> Hi
>>> >>
>>> >> Hi
>>> >>
>>> >> What is the process to become a contributor to drill ?
>>> >>
>>> >> Regards
>>> >> chandan
>>> >>
>>> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>> >>
>>> >> > Suffice it to say that if *you* think it is important enough to
>>> implement
>>> >> > and maintain, then the group shouldn't say naye.  The consensus stuff
>>> >> > should only block things that break something else.  Additive features
>>> >> that
>>> >> > are highly maintainable (or which come with commitments) shouldn't
>>> >> > generally be blocked.
>>> >> >
>>> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
>>> >> > michael.hausenblas@gmail.com> wrote:
>>> >> >
>>> >> > > Good. Feel free to put me down for that, if the group as a whole
>>> thinks
>>> >> > > that (supporting Thrift) makes sense.
>>> >> > >
>>> >> >
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Tomer Shiran
>>> > Director of Product Management | MapR Technologies | 650-804-8657
>>>

Re: Thrift?

Posted by Ryan Rawson <ry...@gmail.com>.
Thanks for that Ted.

Correct - internal wire format doesnt mean 'drill only supports
protobuf encoded data'.

Part of the reason to favor protobuf is that a lot of people in the
broader 'big data' community are building a lot of experience with it.
 Hadoop and HBase both are moving to/moved to protobuf on the wire.
Being able to leverage this expertise is valuable.

There is a JIRA in Hadoop-land where someone had done a deep dive
'bake off' between thrift, protobuf and avro.  The ultimate choice was
protobuf for a number of reasons.  If people want to re-do the
analysis, I'd like to see it in the context of THAT analysis (eg: why
the assumptions there are not the same for Drill)... if anything it'd
give a concrete form to what can be a mire.

For what it's worth, I've had many discussion along these angles with
a variety of people including committers on Thrift, and the consensus
is both are good choices.

-ryan

On Fri, Sep 14, 2012 at 2:31 PM, Ted Dunning <te...@gmail.com> wrote:
> I think that it is important to ask a few questions leading up a decision
> here.
>
> The first is a (rhetorical) show of hands about how many people believe
> that there are no serious performance or expressivity killers when
> comparing alternative serialization frameworks.  As far as I know,
> performance differences are not massive (and protobufs is one of the
> leaders in any case) and the expressivity differences are essentially nil.
>  If somebody feels that there is a serious show-stopper with any option,
> they should speak.
>
> The second is to ask the sense of the community whether they judge progress
> or perfection in this decision is most important to the project.  My guess
> is that almost everybody would prefer to see progress as long as the
> technical choice is not subject to some horrid missing bit.
>
> The final question is whether it is reasonable to go along with protobufs
> given that several very experienced engineers prefer it and would like to
> produce code based on it.  If the first two answers are answered to the
> effect of protobufs is about as good as we will find and that progress
> trumps small differences, then it seems that moving to follow this
> preference of Jason and Ryan for protobufs might be a reasonable thing to
> do.
>
> The question of an internal wire format, btw, does not constrain the
> project relative to external access.  I think it is important to support
> JDBC and ODBC and whatever is in common use for querying.  For external
> access the question is quite different.  Whereas for the internal format
> consensus around a single choice has large benefits, the external format
> choice is nearly the opposite.  For an external format, limiting ourselves
> to a single choice seems like a bad idea and increasing the audience seems
> like a better choice.
>
> On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Hi folks,
>>
>> I just commented on this first JIRA.  Here is my text:
>>
>> This issue has been hashed over a lot in the Hadoop projects. There
>> was work done to compare thrift vs avro vs protobuf. The conclusion
>> was protobuf was the decision to use.
>>
>> Prior to this move, there had been a lot of noise about pluggable RPC
>> transports, and whatnot. It held up adoption of a backwards compatible
>> serialization framework for a long time. The problem ended up being
>> the analysis-paralysis, rather than the specific implementation
>> problem. In other words, the problem was a LACK of implementation than
>> actual REAL problems.
>>
>> Based on this experience, I'd strongly suggest adopting protobuf and
>> moving on. Forget about pluggable RPC implementations, the complexity
>> doesnt deliver benefits. The benefits of protobuf is that its the RPC
>> format for Hadoop and HBase, which allows Drill to draw on the broad
>> experience of those communities who need to implement high performance
>> backwards compatible RPC serialization.
>>
>> ====
>>
>> Expanding a bit, I've looked in to this issue a lot, and there is very
>> few significant concrete reasons to choose protobuf vs thrift.  Tiny
>> percent faster of this, and that, etc.  I'd strongly suggest protobuf
>> for the expanded community.  There is no particular Apache imperative
>> that Apache projects re-use libraries.  Use what makes sense for your
>> project.
>>
>> As regards to Avro, it's a fine serialization format for long term
>> data retention, but the complexities that exist to enable that make it
>> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
>>
>> -ryan
>>
>> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
>> wrote:
>> > We plan to propose the architecture and interfaces in the next couple
>> > weeks, which will make it easy to divide the project into clear building
>> > blocks. At that point it will be easier to start contributing different
>> > data sources, data formats, operators, query languages, etc.
>> >
>> > The contributions are done in the usual Apache way. It's best to open a
>> > JIRA and then post a patch so that others can review and then a committer
>> > can check it in.
>> >
>> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
>> chandanmadhesia@gmail.com
>> >> wrote:
>> >
>> >> Hi
>> >>
>> >> Hi
>> >>
>> >> What is the process to become a contributor to drill ?
>> >>
>> >> Regards
>> >> chandan
>> >>
>> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>> >>
>> >> > Suffice it to say that if *you* think it is important enough to
>> implement
>> >> > and maintain, then the group shouldn't say naye.  The consensus stuff
>> >> > should only block things that break something else.  Additive features
>> >> that
>> >> > are highly maintainable (or which come with commitments) shouldn't
>> >> > generally be blocked.
>> >> >
>> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
>> >> > michael.hausenblas@gmail.com> wrote:
>> >> >
>> >> > > Good. Feel free to put me down for that, if the group as a whole
>> thinks
>> >> > > that (supporting Thrift) makes sense.
>> >> > >
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Tomer Shiran
>> > Director of Product Management | MapR Technologies | 650-804-8657
>>

Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
I think that it is important to ask a few questions leading up a decision
here.

The first is a (rhetorical) show of hands about how many people believe
that there are no serious performance or expressivity killers when
comparing alternative serialization frameworks.  As far as I know,
performance differences are not massive (and protobufs is one of the
leaders in any case) and the expressivity differences are essentially nil.
 If somebody feels that there is a serious show-stopper with any option,
they should speak.

The second is to ask the sense of the community whether they judge progress
or perfection in this decision is most important to the project.  My guess
is that almost everybody would prefer to see progress as long as the
technical choice is not subject to some horrid missing bit.

The final question is whether it is reasonable to go along with protobufs
given that several very experienced engineers prefer it and would like to
produce code based on it.  If the first two answers are answered to the
effect of protobufs is about as good as we will find and that progress
trumps small differences, then it seems that moving to follow this
preference of Jason and Ryan for protobufs might be a reasonable thing to
do.

The question of an internal wire format, btw, does not constrain the
project relative to external access.  I think it is important to support
JDBC and ODBC and whatever is in common use for querying.  For external
access the question is quite different.  Whereas for the internal format
consensus around a single choice has large benefits, the external format
choice is nearly the opposite.  For an external format, limiting ourselves
to a single choice seems like a bad idea and increasing the audience seems
like a better choice.

On Fri, Sep 14, 2012 at 12:44 PM, Ryan Rawson <ry...@gmail.com> wrote:

> Hi folks,
>
> I just commented on this first JIRA.  Here is my text:
>
> This issue has been hashed over a lot in the Hadoop projects. There
> was work done to compare thrift vs avro vs protobuf. The conclusion
> was protobuf was the decision to use.
>
> Prior to this move, there had been a lot of noise about pluggable RPC
> transports, and whatnot. It held up adoption of a backwards compatible
> serialization framework for a long time. The problem ended up being
> the analysis-paralysis, rather than the specific implementation
> problem. In other words, the problem was a LACK of implementation than
> actual REAL problems.
>
> Based on this experience, I'd strongly suggest adopting protobuf and
> moving on. Forget about pluggable RPC implementations, the complexity
> doesnt deliver benefits. The benefits of protobuf is that its the RPC
> format for Hadoop and HBase, which allows Drill to draw on the broad
> experience of those communities who need to implement high performance
> backwards compatible RPC serialization.
>
> ====
>
> Expanding a bit, I've looked in to this issue a lot, and there is very
> few significant concrete reasons to choose protobuf vs thrift.  Tiny
> percent faster of this, and that, etc.  I'd strongly suggest protobuf
> for the expanded community.  There is no particular Apache imperative
> that Apache projects re-use libraries.  Use what makes sense for your
> project.
>
> As regards to Avro, it's a fine serialization format for long term
> data retention, but the complexities that exist to enable that make it
> non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.
>
> -ryan
>
> On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com>
> wrote:
> > We plan to propose the architecture and interfaces in the next couple
> > weeks, which will make it easy to divide the project into clear building
> > blocks. At that point it will be easier to start contributing different
> > data sources, data formats, operators, query languages, etc.
> >
> > The contributions are done in the usual Apache way. It's best to open a
> > JIRA and then post a patch so that others can review and then a committer
> > can check it in.
> >
> > On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <
> chandanmadhesia@gmail.com
> >> wrote:
> >
> >> Hi
> >>
> >> Hi
> >>
> >> What is the process to become a contributor to drill ?
> >>
> >> Regards
> >> chandan
> >>
> >> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com>
> wrote:
> >>
> >> > Suffice it to say that if *you* think it is important enough to
> implement
> >> > and maintain, then the group shouldn't say naye.  The consensus stuff
> >> > should only block things that break something else.  Additive features
> >> that
> >> > are highly maintainable (or which come with commitments) shouldn't
> >> > generally be blocked.
> >> >
> >> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> >> > michael.hausenblas@gmail.com> wrote:
> >> >
> >> > > Good. Feel free to put me down for that, if the group as a whole
> thinks
> >> > > that (supporting Thrift) makes sense.
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> > Tomer Shiran
> > Director of Product Management | MapR Technologies | 650-804-8657
>

Re: Thrift?

Posted by Ryan Rawson <ry...@gmail.com>.
Hi folks,

I just commented on this first JIRA.  Here is my text:

This issue has been hashed over a lot in the Hadoop projects. There
was work done to compare thrift vs avro vs protobuf. The conclusion
was protobuf was the decision to use.

Prior to this move, there had been a lot of noise about pluggable RPC
transports, and whatnot. It held up adoption of a backwards compatible
serialization framework for a long time. The problem ended up being
the analysis-paralysis, rather than the specific implementation
problem. In other words, the problem was a LACK of implementation than
actual REAL problems.

Based on this experience, I'd strongly suggest adopting protobuf and
moving on. Forget about pluggable RPC implementations, the complexity
doesnt deliver benefits. The benefits of protobuf is that its the RPC
format for Hadoop and HBase, which allows Drill to draw on the broad
experience of those communities who need to implement high performance
backwards compatible RPC serialization.

====

Expanding a bit, I've looked in to this issue a lot, and there is very
few significant concrete reasons to choose protobuf vs thrift.  Tiny
percent faster of this, and that, etc.  I'd strongly suggest protobuf
for the expanded community.  There is no particular Apache imperative
that Apache projects re-use libraries.  Use what makes sense for your
project.

As regards to Avro, it's a fine serialization format for long term
data retention, but the complexities that exist to enable that make it
non-ideal for an RPC.  I know of no one who uses AvroRPC in any form.

-ryan

On Tue, Sep 4, 2012 at 12:30 PM, Tomer Shiran <ts...@maprtech.com> wrote:
> We plan to propose the architecture and interfaces in the next couple
> weeks, which will make it easy to divide the project into clear building
> blocks. At that point it will be easier to start contributing different
> data sources, data formats, operators, query languages, etc.
>
> The contributions are done in the usual Apache way. It's best to open a
> JIRA and then post a patch so that others can review and then a committer
> can check it in.
>
> On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <chandanmadhesia@gmail.com
>> wrote:
>
>> Hi
>>
>> Hi
>>
>> What is the process to become a contributor to drill ?
>>
>> Regards
>> chandan
>>
>> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com> wrote:
>>
>> > Suffice it to say that if *you* think it is important enough to implement
>> > and maintain, then the group shouldn't say naye.  The consensus stuff
>> > should only block things that break something else.  Additive features
>> that
>> > are highly maintainable (or which come with commitments) shouldn't
>> > generally be blocked.
>> >
>> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
>> > michael.hausenblas@gmail.com> wrote:
>> >
>> > > Good. Feel free to put me down for that, if the group as a whole thinks
>> > > that (supporting Thrift) makes sense.
>> > >
>> >
>>
>
>
>
> --
> Tomer Shiran
> Director of Product Management | MapR Technologies | 650-804-8657

Re: Thrift?

Posted by Tomer Shiran <ts...@maprtech.com>.
We plan to propose the architecture and interfaces in the next couple
weeks, which will make it easy to divide the project into clear building
blocks. At that point it will be easier to start contributing different
data sources, data formats, operators, query languages, etc.

The contributions are done in the usual Apache way. It's best to open a
JIRA and then post a patch so that others can review and then a committer
can check it in.

On Tue, Sep 4, 2012 at 12:23 PM, Chandan Madhesia <chandanmadhesia@gmail.com
> wrote:

> Hi
>
> Hi
>
> What is the process to become a contributor to drill ?
>
> Regards
> chandan
>
> On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com> wrote:
>
> > Suffice it to say that if *you* think it is important enough to implement
> > and maintain, then the group shouldn't say naye.  The consensus stuff
> > should only block things that break something else.  Additive features
> that
> > are highly maintainable (or which come with commitments) shouldn't
> > generally be blocked.
> >
> > On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> > michael.hausenblas@gmail.com> wrote:
> >
> > > Good. Feel free to put me down for that, if the group as a whole thinks
> > > that (supporting Thrift) makes sense.
> > >
> >
>



-- 
Tomer Shiran
Director of Product Management | MapR Technologies | 650-804-8657

Re: Thrift?

Posted by Chandan Madhesia <ch...@gmail.com>.
Hi

Hi

What is the process to become a contributor to drill ?

Regards
chandan

On Tue, Sep 4, 2012 at 9:51 PM, Ted Dunning <te...@gmail.com> wrote:

> Suffice it to say that if *you* think it is important enough to implement
> and maintain, then the group shouldn't say naye.  The consensus stuff
> should only block things that break something else.  Additive features that
> are highly maintainable (or which come with commitments) shouldn't
> generally be blocked.
>
> On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
> michael.hausenblas@gmail.com> wrote:
>
> > Good. Feel free to put me down for that, if the group as a whole thinks
> > that (supporting Thrift) makes sense.
> >
>

Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
Suffice it to say that if *you* think it is important enough to implement
and maintain, then the group shouldn't say naye.  The consensus stuff
should only block things that break something else.  Additive features that
are highly maintainable (or which come with commitments) shouldn't
generally be blocked.

On Tue, Sep 4, 2012 at 9:14 AM, Michael Hausenblas <
michael.hausenblas@gmail.com> wrote:

> Good. Feel free to put me down for that, if the group as a whole thinks
> that (supporting Thrift) makes sense.
>

Re: Thrift?

Posted by Michael Hausenblas <mi...@gmail.com>.
> As such, if you have an itch to have a Thrift based wire protocol, that is
> a great idea and it should be relatively easy to build one.

Good. Feel free to put me down for that, if the group as a whole thinks that (supporting Thrift) makes sense.


> It should also be relatively opaque to the user what wire protocol is in use.


Exactly. And this should be made explicit in the design considerations and requirements.

Cheers,
	   Michael

--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/

On 4 Sep 2012, at 17:08, Ted Dunning wrote:

> I think that Thrift would make a fine remote API protocol.  And I sincerely
> hope that we don't lock Drill into one data representation or one RPC wire
> format.  API != wire protocol, after all.
> 
> As such, if you have an itch to have a Thrift based wire protocol, that is
> a great idea and it should be relatively easy to build one.
> 
> It should also be relatively opaque to the user what wire protocol is in
> use.
> 
> On Tue, Sep 4, 2012 at 8:28 AM, lesliewang <wq...@hotmail.com> wrote:
> 
>> 
>> Good point. They are quite similar.
>> I guess it is mainly because Dremel is developed by Google, and explain
>> how to use Protocol Buffer in paper. Thus, Protocol Buffers are adopted in
>> Drill by default. :-)
>> 
>>> From: michael.hausenblas@gmail.com
>>> Subject: Thrift?
>>> Date: Tue, 4 Sep 2012 11:16:11 +0100
>>> To: drill-dev@incubator.apache.org
>>> 
>>> All,
>>> 
>>> I'm wondering: Protocol Buffers (and Avro) is mentioned a couple of
>> times both in the proposal and in the git README but no mention of Apache
>> Thrift [1] at all.
>>> 
>>> Based on uptake and being a top-level Apache project, I think we might
>> want to consider re-using/basing Drill on Thrift, or, at least, mention it
>> as an option?
>>> 
>>> Cheers,
>>>         Michael
>>> 
>>> [1] http://thrift.apache.org/
>>> 
>>> --
>>> Michael Hausenblas
>>> Ireland, Europe
>>> http://mhausenblas.info/
>>> 
>> 
>> 


Re: Thrift?

Posted by Ted Dunning <te...@gmail.com>.
I think that Thrift would make a fine remote API protocol.  And I sincerely
hope that we don't lock Drill into one data representation or one RPC wire
format.  API != wire protocol, after all.

As such, if you have an itch to have a Thrift based wire protocol, that is
a great idea and it should be relatively easy to build one.

It should also be relatively opaque to the user what wire protocol is in
use.

On Tue, Sep 4, 2012 at 8:28 AM, lesliewang <wq...@hotmail.com> wrote:

>
> Good point. They are quite similar.
> I guess it is mainly because Dremel is developed by Google, and explain
> how to use Protocol Buffer in paper. Thus, Protocol Buffers are adopted in
> Drill by default. :-)
>
> > From: michael.hausenblas@gmail.com
> > Subject: Thrift?
> > Date: Tue, 4 Sep 2012 11:16:11 +0100
> > To: drill-dev@incubator.apache.org
> >
> > All,
> >
> > I'm wondering: Protocol Buffers (and Avro) is mentioned a couple of
> times both in the proposal and in the git README but no mention of Apache
> Thrift [1] at all.
> >
> > Based on uptake and being a top-level Apache project, I think we might
> want to consider re-using/basing Drill on Thrift, or, at least, mention it
> as an option?
> >
> > Cheers,
> >          Michael
> >
> > [1] http://thrift.apache.org/
> >
> > --
> > Michael Hausenblas
> > Ireland, Europe
> > http://mhausenblas.info/
> >
>
>

RE: Thrift?

Posted by lesliewang <wq...@hotmail.com>.
Good point. They are quite similar. 
I guess it is mainly because Dremel is developed by Google, and explain how to use Protocol Buffer in paper. Thus, Protocol Buffers are adopted in Drill by default. :-)

> From: michael.hausenblas@gmail.com
> Subject: Thrift?
> Date: Tue, 4 Sep 2012 11:16:11 +0100
> To: drill-dev@incubator.apache.org
> 
> All,
> 
> I'm wondering: Protocol Buffers (and Avro) is mentioned a couple of times both in the proposal and in the git README but no mention of Apache Thrift [1] at all.
> 
> Based on uptake and being a top-level Apache project, I think we might want to consider re-using/basing Drill on Thrift, or, at least, mention it as an option?
> 
> Cheers,
> 	   Michael
> 
> [1] http://thrift.apache.org/
> 
> --
> Michael Hausenblas
> Ireland, Europe
> http://mhausenblas.info/
>