Proposal for design of the traits for the heron scala API--Looking for feedback from dev community

[Saikat Kanjilal <sx...@hotmail.com>]

Folks,
@karthikz asked me to publish this on the dev list, so here goes:

I've taken an initial stab below at getting the deisgn doc for the scala API in shape:  https://docs.google.com/document/d/1vIL4hVC4SwYU5YkP9cFSn6poJ0M1_XROrB-Kl50xDs4/edit . and in doing so I've added the designs around the traits builder, source,sink,streamlet
[https://lh5.googleusercontent.com/ojQSM8oogNBO7ST645PYSkaXilzntJr_vPfc_rPcfsRAMugB0ZRKj92qiXPqxyPBwHzBJA=w1200-h630-p]<https://docs.google.com/document/d/1vIL4hVC4SwYU5YkP9cFSn6poJ0M1_XROrB-Kl50xDs4/edit>

API Design for Heron Scala Port<https://docs.google.com/document/d/1vIL4hVC4SwYU5YkP9cFSn6poJ0M1_XROrB-Kl50xDs4/edit>
API Design for Heron Scala Port Introduction This document will outline all of the api’s associated with the scala port for heron. The design will consist of a set of interfaces , I will build the codebase after every interface conversion so that the design actually reflects the port which b...
docs.google.com


For now I've tackled only the source, sink, streamlet traits.  Once we can agree on the design for these we can tackle the higher level traits that depend on these.

The biggest open question in my mind is how we represent Java serializable functions that need to passed into some of the traits, I will be researching methodologies to do this as folks add comments to the doc.  Given that the passed in functions are passed in the options we have are . 1) make the functions extend Serializable 2) keep them pure scala functions and use case classes to implement serializability transparently


Looking forward to hearing back.