You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Jacques Nadeau <ja...@gmail.com> on 2013/01/15 00:56:19 UTC
First pass at a reference interpreter
I've been pulling together a reference logical plan interpreter. I'm
working with Ted to get it inside the Drill sandbox. For now, you can find
it on my repo at https://github.com/jacques-n/incubator-drill (prototype
branch)
The goals of the reference interpreter are:
- To provide a simple way to run a Logical Plan against some sample data
and get back the expected result
- Allow work to start on the parsers while we scale up the performance
and capabilities of the execution engine and optimizer.
- Allow evaluation work on particular technical approaches such as
exploring the impact of hierarchical and schema less data on query
evaluation.
These goals do not include performance, memory handling, or
efficiency. Currently,
the interpreter is a single node/thread process. This will change shortly
so that it also run as a clustered process.
The entry point is inside the /sandbox/prototype/exec/ref module:
org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example program
utilizes two resources: simple-plan.json and donuts.json and outputs data
to /opt/data/out.json.
Some of things that 'work'.
- Read/write basic json.
- ROPs (reference operators): Filter, Transform, Group, Aggregate
(simple), Order, Union.
- Example aggregate and basic functions including sum, count, multiply,
add, compare, equals.
Basic glossary/concepts (we'll get this on the wiki/javadocs):
- LOP: Logical Operator. An implementation agnostic data flow operator
utilized by the Logical Plan.
- ROP: Reference Operator: A reference operator implementation that
pairs with a LOP.
- FunctionDefinition: A definition of a particular function. Describes
a set of aliases, an allowable set of input arguments and an interface that
will attempt to determine output type.
- BasicEvaluator: An implementation of a particular non-aggregate
expression. Receives a record pointer at creation time. Returns a
DataValue.
- AggregateEvaluator: An implementation of a particular aggregating
function. Is provided a record pointer at creation time. Expects regular
calls to addRecord() followed by a call to eval() which provides the
aggregate value.
- DataValue: A pointer to a particular data value. Implementation
classes includes things like ScalarLong, ScalarBytes, SimpleMapValue and
SimpleArrayValue.
The standard record iterator utilized between each ROP utilizes the
org.apache.drill.exec.ref.RecordIterator interface. This is somewhat
inspired by the AttributeSource concepts from within the Lucene project.
(I'm planning to extend these concepts all the way to the individual
DataValues.)
My next goals are to add tests, finish adding ROPs, add local and remote
exchange nodes (parallelization), add a bunch of documentation and extract
out the Execution plan as a separate intermediate representation.
It needs a lot more evaluators to be a true reference interpreter (as well
as the rest of the ROPs). The existing ones can be utilized as prototypes.
Anyone interested in ripping through a bunch of additional evaluators and
associated FunctionDefinitions?
Re: First pass at a reference interpreter
Posted by Andrew Psaltis <An...@Webtrends.com>.
This is great to see. Can't wait to start using and contributing.
In good health,
Andrew
Sent from my GPU powered iPhone
On Jan 14, 2013, at 16:56, "Jacques Nadeau" <ja...@gmail.com> wrote:
> I've been pulling together a reference logical plan interpreter. I'm
> working with Ted to get it inside the Drill sandbox. For now, you can find
> it on my repo at https://github.com/jacques-n/incubator-drill (prototype
> branch)
>
>
>
> The goals of the reference interpreter are:
>
>
> - To provide a simple way to run a Logical Plan against some sample data
> and get back the expected result
> - Allow work to start on the parsers while we scale up the performance
> and capabilities of the execution engine and optimizer.
> - Allow evaluation work on particular technical approaches such as
> exploring the impact of hierarchical and schema less data on query
> evaluation.
>
> These goals do not include performance, memory handling, or
> efficiency. Currently,
> the interpreter is a single node/thread process. This will change shortly
> so that it also run as a clustered process.
>
> The entry point is inside the /sandbox/prototype/exec/ref module:
> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example program
> utilizes two resources: simple-plan.json and donuts.json and outputs data
> to /opt/data/out.json.
>
>
> Some of things that 'work'.
>
>
> - Read/write basic json.
> - ROPs (reference operators): Filter, Transform, Group, Aggregate
> (simple), Order, Union.
> - Example aggregate and basic functions including sum, count, multiply,
> add, compare, equals.
>
> Basic glossary/concepts (we'll get this on the wiki/javadocs):
>
>
> - LOP: Logical Operator. An implementation agnostic data flow operator
> utilized by the Logical Plan.
> - ROP: Reference Operator: A reference operator implementation that
> pairs with a LOP.
> - FunctionDefinition: A definition of a particular function. Describes
> a set of aliases, an allowable set of input arguments and an interface that
> will attempt to determine output type.
> - BasicEvaluator: An implementation of a particular non-aggregate
> expression. Receives a record pointer at creation time. Returns a
> DataValue.
> - AggregateEvaluator: An implementation of a particular aggregating
> function. Is provided a record pointer at creation time. Expects regular
> calls to addRecord() followed by a call to eval() which provides the
> aggregate value.
> - DataValue: A pointer to a particular data value. Implementation
> classes includes things like ScalarLong, ScalarBytes, SimpleMapValue and
> SimpleArrayValue.
>
> The standard record iterator utilized between each ROP utilizes the
> org.apache.drill.exec.ref.RecordIterator interface. This is somewhat
> inspired by the AttributeSource concepts from within the Lucene project.
> (I'm planning to extend these concepts all the way to the individual
> DataValues.)
>
>
>
> My next goals are to add tests, finish adding ROPs, add local and remote
> exchange nodes (parallelization), add a bunch of documentation and extract
> out the Execution plan as a separate intermediate representation.
>
>
>
> It needs a lot more evaluators to be a true reference interpreter (as well
> as the rest of the ROPs). The existing ones can be utilized as prototypes.
> Anyone interested in ripping through a bunch of additional evaluators and
> associated FunctionDefinitions?
Re: First pass at a reference interpreter
Posted by Jacques Nadeau <ja...@gmail.com>.
People can update the simple_plan.json to point to they want the output to
go. A good feature would be to provide a wrapping interface that is a pipe
sink to return to console.
On Tue, Jan 15, 2013 at 7:22 PM, Ted Dunning <te...@gmail.com> wrote:
> I would suggest not putting the output in a fixed path. Most users won't
> have write permission by default.
>
> On Tue, Jan 15, 2013 at 9:26 AM, Jacques Nadeau <jacques.drill@gmail.com
> >wrote:
>
> > Command line running:
> >
> > cd sandbox/prototype/exec/ref
> > mvn exec:java
> > -Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter" && cat
> > /opt/data/out.json
> > (note, the cat is required because the only current sink is a json writer
> > and the plan currently states to write to this location.)
> >
> > Data input and query files are at:
> > sandbox/prototype/exec/ref/src/test/resources/
> > >donuts.json is a small sample data file.
> > >simple_plan.json is a logical plan.
> >
> >
> >
> > On Mon, Jan 14, 2013 at 5:07 PM, Michael Hausenblas <
> > michael.hausenblas@gmail.com> wrote:
> >
> > >
> > > > Given that Java6 is public updates EOL next month, I've targeted
> Java7.
> > > > You probably need to update your JDK.
> > >
> > > Good point. I suppose this should be in big red blinking letters on top
> > of
> > > the installation instructions.
> > >
> > > 'mvn verify' is a success now for me ... in case anyone is on MacOS 10,
> > > I've used the JDK from [1] and works like a charm.
> > >
> > > Cheers,
> > > Michael
> > >
> > > [1]
> > >
> >
> http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
> > >
> > > --
> > > Michael Hausenblas
> > > Ireland, Europe
> > > http://mhausenblas.info/
> > >
> > > On 14 Jan 2013, at 16:37, Jacques Nadeau <ja...@gmail.com>
> > wrote:
> > >
> > > > Given that Java6 is public updates EOL next month, I've targeted
> Java7.
> > > > You probably need to update your JDK.
> > > >
> > > > Jacques
> > > >
> > > > On Mon, Jan 14, 2013 at 4:31 PM, Michael Hausenblas <
> > > > michael.hausenblas@gmail.com> wrote:
> > > >
> > > >>
> > > >>> Be careful, there are many sharp edges.
> > > >>>
> > > >>> :)
> > > >>
> > > >>
> > > >> Alright ;)
> > > >>
> > > >> So here we go: 'mvn validate' works fine for me but then ...
> > > >>
> > > >> $ mvn verify
> > > >>
> > > >> [INFO] prototype-parent .................................. SUCCESS
> > > [7.166s]
> > > >> [INFO] common ............................................ FAILURE
> > > >> [17.775s]
> > > >> [INFO] contrib-parent .................................... SKIPPED
> > > >> [INFO] pom ............................................... SKIPPED
> > > >> [INFO] exec .............................................. SKIPPED
> > > >> [INFO] java-exec ......................................... SKIPPED
> > > >> [INFO] Logical Plan Execution Reference Implementation ... SKIPPED
> > > >> [INFO] planner ........................................... SKIPPED
> > > >> [INFO] sqlparser ......................................... SKIPPED
> > > >>
> > > >>
> > > >> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> > > execute
> > > >> goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
> > > >> (default-compile) on project common: Compilation failure
> > > >> Failure executing javac, but could not parse the error:
> > > >> javac: invalid target release: 1.7
> > > >> Usage: javac <options> <source files>
> > > >> use -help for a list of possible options
> > > >>
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> > > >> at
> > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
> > > >> at
> org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> > > >> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> > > >> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> > > >> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > >> at java.lang.reflect.Method.invoke(Method.java:597)
> > > >> at
> > > >>
> > >
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> > > >> at
> > > >>
> > >
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> > > >> at
> > > >>
> > >
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> > > >> at
> > > >>
> > >
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> > > >> Caused by: org.apache.maven.plugin.CompilationFailureException:
> > > >> Compilation failure
> > > >> Failure executing javac, but could not parse the error:
> > > >> javac: invalid target release: 1.7
> > > >> Usage: javac <options> <source files>
> > > >> use -help for a list of possible options
> > > >>
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
> > > >> at
> > > >> org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114)
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> > > >> at
> > > >>
> > >
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> > > >>
> > > >>
> > > >>
> > > >> Cheers,
> > > >> Michael
> > > >>
> > > >> --
> > > >> Michael Hausenblas
> > > >> Ireland, Europe
> > > >> http://mhausenblas.info/
> > > >>
> > > >> On 14 Jan 2013, at 16:06, Jacques Nadeau <ja...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Be careful, there are many sharp edges.
> > > >>>
> > > >>> :)
> > > >>>
> > > >>> On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas <
> > > >>> michael.hausenblas@gmail.com> wrote:
> > > >>>
> > > >>>>
> > > >>>> Cool stuff, Jacques - will give it a shot ASAP!
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Michael
> > > >>>>
> > > >>>> --
> > > >>>> Michael Hausenblas
> > > >>>> Ireland, Europe
> > > >>>> http://mhausenblas.info/
> > > >>>>
> > > >>>> On 14 Jan 2013, at 15:56, Jacques Nadeau <jacques.drill@gmail.com
> >
> > > >> wrote:
> > > >>>>
> > > >>>>> I've been pulling together a reference logical plan interpreter.
> > I'm
> > > >>>>> working with Ted to get it inside the Drill sandbox. For now,
> you
> > > can
> > > >>>> find
> > > >>>>> it on my repo at
> > > https://github.com/jacques-n/incubator-drill(prototype
> > > >>>>> branch)
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> The goals of the reference interpreter are:
> > > >>>>>
> > > >>>>>
> > > >>>>> - To provide a simple way to run a Logical Plan against some
> sample
> > > >>>> data
> > > >>>>> and get back the expected result
> > > >>>>> - Allow work to start on the parsers while we scale up the
> > > performance
> > > >>>>> and capabilities of the execution engine and optimizer.
> > > >>>>> - Allow evaluation work on particular technical approaches such
> as
> > > >>>>> exploring the impact of hierarchical and schema less data on
> query
> > > >>>>> evaluation.
> > > >>>>>
> > > >>>>> These goals do not include performance, memory handling, or
> > > >>>>> efficiency. Currently,
> > > >>>>> the interpreter is a single node/thread process. This will
> change
> > > >>>> shortly
> > > >>>>> so that it also run as a clustered process.
> > > >>>>>
> > > >>>>> The entry point is inside the /sandbox/prototype/exec/ref module:
> > > >>>>> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The
> example
> > > >>>> program
> > > >>>>> utilizes two resources: simple-plan.json and donuts.json and
> > outputs
> > > >> data
> > > >>>>> to /opt/data/out.json.
> > > >>>>>
> > > >>>>>
> > > >>>>> Some of things that 'work'.
> > > >>>>>
> > > >>>>>
> > > >>>>> - Read/write basic json.
> > > >>>>> - ROPs (reference operators): Filter, Transform, Group, Aggregate
> > > >>>>> (simple), Order, Union.
> > > >>>>> - Example aggregate and basic functions including sum, count,
> > > >> multiply,
> > > >>>>> add, compare, equals.
> > > >>>>>
> > > >>>>> Basic glossary/concepts (we'll get this on the wiki/javadocs):
> > > >>>>>
> > > >>>>>
> > > >>>>> - LOP: Logical Operator. An implementation agnostic data flow
> > > >> operator
> > > >>>>> utilized by the Logical Plan.
> > > >>>>> - ROP: Reference Operator: A reference operator implementation
> that
> > > >>>>> pairs with a LOP.
> > > >>>>> - FunctionDefinition: A definition of a particular function.
> > > >> Describes
> > > >>>>> a set of aliases, an allowable set of input arguments and an
> > > interface
> > > >>>> that
> > > >>>>> will attempt to determine output type.
> > > >>>>> - BasicEvaluator: An implementation of a particular non-aggregate
> > > >>>>> expression. Receives a record pointer at creation time. Returns
> a
> > > >>>>> DataValue.
> > > >>>>> - AggregateEvaluator: An implementation of a particular
> aggregating
> > > >>>>> function. Is provided a record pointer at creation time.
> Expects
> > > >>>> regular
> > > >>>>> calls to addRecord() followed by a call to eval() which provides
> > the
> > > >>>>> aggregate value.
> > > >>>>> - DataValue: A pointer to a particular data value.
> Implementation
> > > >>>>> classes includes things like ScalarLong, ScalarBytes,
> > SimpleMapValue
> > > >>>> and
> > > >>>>> SimpleArrayValue.
> > > >>>>>
> > > >>>>> The standard record iterator utilized between each ROP utilizes
> the
> > > >>>>> org.apache.drill.exec.ref.RecordIterator interface. This is
> > somewhat
> > > >>>>> inspired by the AttributeSource concepts from within the Lucene
> > > >> project.
> > > >>>>> (I'm planning to extend these concepts all the way to the
> > individual
> > > >>>>> DataValues.)
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> My next goals are to add tests, finish adding ROPs, add local and
> > > >> remote
> > > >>>>> exchange nodes (parallelization), add a bunch of documentation
> and
> > > >>>> extract
> > > >>>>> out the Execution plan as a separate intermediate representation.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> It needs a lot more evaluators to be a true reference interpreter
> > (as
> > > >>>> well
> > > >>>>> as the rest of the ROPs). The existing ones can be utilized as
> > > >>>> prototypes.
> > > >>>>> Anyone interested in ripping through a bunch of additional
> > evaluators
> > > >> and
> > > >>>>> associated FunctionDefinitions?
> > > >>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>
Re: First pass at a reference interpreter
Posted by Ted Dunning <te...@gmail.com>.
I would suggest not putting the output in a fixed path. Most users won't
have write permission by default.
On Tue, Jan 15, 2013 at 9:26 AM, Jacques Nadeau <ja...@gmail.com>wrote:
> Command line running:
>
> cd sandbox/prototype/exec/ref
> mvn exec:java
> -Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter" && cat
> /opt/data/out.json
> (note, the cat is required because the only current sink is a json writer
> and the plan currently states to write to this location.)
>
> Data input and query files are at:
> sandbox/prototype/exec/ref/src/test/resources/
> >donuts.json is a small sample data file.
> >simple_plan.json is a logical plan.
>
>
>
> On Mon, Jan 14, 2013 at 5:07 PM, Michael Hausenblas <
> michael.hausenblas@gmail.com> wrote:
>
> >
> > > Given that Java6 is public updates EOL next month, I've targeted Java7.
> > > You probably need to update your JDK.
> >
> > Good point. I suppose this should be in big red blinking letters on top
> of
> > the installation instructions.
> >
> > 'mvn verify' is a success now for me ... in case anyone is on MacOS 10,
> > I've used the JDK from [1] and works like a charm.
> >
> > Cheers,
> > Michael
> >
> > [1]
> >
> http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
> >
> > --
> > Michael Hausenblas
> > Ireland, Europe
> > http://mhausenblas.info/
> >
> > On 14 Jan 2013, at 16:37, Jacques Nadeau <ja...@gmail.com>
> wrote:
> >
> > > Given that Java6 is public updates EOL next month, I've targeted Java7.
> > > You probably need to update your JDK.
> > >
> > > Jacques
> > >
> > > On Mon, Jan 14, 2013 at 4:31 PM, Michael Hausenblas <
> > > michael.hausenblas@gmail.com> wrote:
> > >
> > >>
> > >>> Be careful, there are many sharp edges.
> > >>>
> > >>> :)
> > >>
> > >>
> > >> Alright ;)
> > >>
> > >> So here we go: 'mvn validate' works fine for me but then ...
> > >>
> > >> $ mvn verify
> > >>
> > >> [INFO] prototype-parent .................................. SUCCESS
> > [7.166s]
> > >> [INFO] common ............................................ FAILURE
> > >> [17.775s]
> > >> [INFO] contrib-parent .................................... SKIPPED
> > >> [INFO] pom ............................................... SKIPPED
> > >> [INFO] exec .............................................. SKIPPED
> > >> [INFO] java-exec ......................................... SKIPPED
> > >> [INFO] Logical Plan Execution Reference Implementation ... SKIPPED
> > >> [INFO] planner ........................................... SKIPPED
> > >> [INFO] sqlparser ......................................... SKIPPED
> > >>
> > >>
> > >> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> > execute
> > >> goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
> > >> (default-compile) on project common: Compilation failure
> > >> Failure executing javac, but could not parse the error:
> > >> javac: invalid target release: 1.7
> > >> Usage: javac <options> <source files>
> > >> use -help for a list of possible options
> > >>
> > >>
> > >> at
> > >>
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
> > >> at
> > >>
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> > >> at
> > >>
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> > >> at
> > >>
> >
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> > >> at
> > >>
> >
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> > >> at
> > >>
> >
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> > >> at
> > >>
> >
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> > >> at
> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
> > >> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> > >> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> > >> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> > >> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > >> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >> at java.lang.reflect.Method.invoke(Method.java:597)
> > >> at
> > >>
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> > >> at
> > >>
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> > >> at
> > >>
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> > >> at
> > >>
> > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> > >> Caused by: org.apache.maven.plugin.CompilationFailureException:
> > >> Compilation failure
> > >> Failure executing javac, but could not parse the error:
> > >> javac: invalid target release: 1.7
> > >> Usage: javac <options> <source files>
> > >> use -help for a list of possible options
> > >>
> > >>
> > >> at
> > >>
> >
> org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
> > >> at
> > >> org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114)
> > >> at
> > >>
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> > >> at
> > >>
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> > >>
> > >>
> > >>
> > >> Cheers,
> > >> Michael
> > >>
> > >> --
> > >> Michael Hausenblas
> > >> Ireland, Europe
> > >> http://mhausenblas.info/
> > >>
> > >> On 14 Jan 2013, at 16:06, Jacques Nadeau <ja...@gmail.com>
> > wrote:
> > >>
> > >>> Be careful, there are many sharp edges.
> > >>>
> > >>> :)
> > >>>
> > >>> On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas <
> > >>> michael.hausenblas@gmail.com> wrote:
> > >>>
> > >>>>
> > >>>> Cool stuff, Jacques - will give it a shot ASAP!
> > >>>>
> > >>>> Cheers,
> > >>>> Michael
> > >>>>
> > >>>> --
> > >>>> Michael Hausenblas
> > >>>> Ireland, Europe
> > >>>> http://mhausenblas.info/
> > >>>>
> > >>>> On 14 Jan 2013, at 15:56, Jacques Nadeau <ja...@gmail.com>
> > >> wrote:
> > >>>>
> > >>>>> I've been pulling together a reference logical plan interpreter.
> I'm
> > >>>>> working with Ted to get it inside the Drill sandbox. For now, you
> > can
> > >>>> find
> > >>>>> it on my repo at
> > https://github.com/jacques-n/incubator-drill(prototype
> > >>>>> branch)
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> The goals of the reference interpreter are:
> > >>>>>
> > >>>>>
> > >>>>> - To provide a simple way to run a Logical Plan against some sample
> > >>>> data
> > >>>>> and get back the expected result
> > >>>>> - Allow work to start on the parsers while we scale up the
> > performance
> > >>>>> and capabilities of the execution engine and optimizer.
> > >>>>> - Allow evaluation work on particular technical approaches such as
> > >>>>> exploring the impact of hierarchical and schema less data on query
> > >>>>> evaluation.
> > >>>>>
> > >>>>> These goals do not include performance, memory handling, or
> > >>>>> efficiency. Currently,
> > >>>>> the interpreter is a single node/thread process. This will change
> > >>>> shortly
> > >>>>> so that it also run as a clustered process.
> > >>>>>
> > >>>>> The entry point is inside the /sandbox/prototype/exec/ref module:
> > >>>>> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example
> > >>>> program
> > >>>>> utilizes two resources: simple-plan.json and donuts.json and
> outputs
> > >> data
> > >>>>> to /opt/data/out.json.
> > >>>>>
> > >>>>>
> > >>>>> Some of things that 'work'.
> > >>>>>
> > >>>>>
> > >>>>> - Read/write basic json.
> > >>>>> - ROPs (reference operators): Filter, Transform, Group, Aggregate
> > >>>>> (simple), Order, Union.
> > >>>>> - Example aggregate and basic functions including sum, count,
> > >> multiply,
> > >>>>> add, compare, equals.
> > >>>>>
> > >>>>> Basic glossary/concepts (we'll get this on the wiki/javadocs):
> > >>>>>
> > >>>>>
> > >>>>> - LOP: Logical Operator. An implementation agnostic data flow
> > >> operator
> > >>>>> utilized by the Logical Plan.
> > >>>>> - ROP: Reference Operator: A reference operator implementation that
> > >>>>> pairs with a LOP.
> > >>>>> - FunctionDefinition: A definition of a particular function.
> > >> Describes
> > >>>>> a set of aliases, an allowable set of input arguments and an
> > interface
> > >>>> that
> > >>>>> will attempt to determine output type.
> > >>>>> - BasicEvaluator: An implementation of a particular non-aggregate
> > >>>>> expression. Receives a record pointer at creation time. Returns a
> > >>>>> DataValue.
> > >>>>> - AggregateEvaluator: An implementation of a particular aggregating
> > >>>>> function. Is provided a record pointer at creation time. Expects
> > >>>> regular
> > >>>>> calls to addRecord() followed by a call to eval() which provides
> the
> > >>>>> aggregate value.
> > >>>>> - DataValue: A pointer to a particular data value. Implementation
> > >>>>> classes includes things like ScalarLong, ScalarBytes,
> SimpleMapValue
> > >>>> and
> > >>>>> SimpleArrayValue.
> > >>>>>
> > >>>>> The standard record iterator utilized between each ROP utilizes the
> > >>>>> org.apache.drill.exec.ref.RecordIterator interface. This is
> somewhat
> > >>>>> inspired by the AttributeSource concepts from within the Lucene
> > >> project.
> > >>>>> (I'm planning to extend these concepts all the way to the
> individual
> > >>>>> DataValues.)
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> My next goals are to add tests, finish adding ROPs, add local and
> > >> remote
> > >>>>> exchange nodes (parallelization), add a bunch of documentation and
> > >>>> extract
> > >>>>> out the Execution plan as a separate intermediate representation.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> It needs a lot more evaluators to be a true reference interpreter
> (as
> > >>>> well
> > >>>>> as the rest of the ROPs). The existing ones can be utilized as
> > >>>> prototypes.
> > >>>>> Anyone interested in ripping through a bunch of additional
> evaluators
> > >> and
> > >>>>> associated FunctionDefinitions?
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: First pass at a reference interpreter
Posted by Jacques Nadeau <ja...@gmail.com>.
Command line running:
cd sandbox/prototype/exec/ref
mvn exec:java
-Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter" && cat
/opt/data/out.json
(note, the cat is required because the only current sink is a json writer
and the plan currently states to write to this location.)
Data input and query files are at:
sandbox/prototype/exec/ref/src/test/resources/
>donuts.json is a small sample data file.
>simple_plan.json is a logical plan.
On Mon, Jan 14, 2013 at 5:07 PM, Michael Hausenblas <
michael.hausenblas@gmail.com> wrote:
>
> > Given that Java6 is public updates EOL next month, I've targeted Java7.
> > You probably need to update your JDK.
>
> Good point. I suppose this should be in big red blinking letters on top of
> the installation instructions.
>
> 'mvn verify' is a success now for me ... in case anyone is on MacOS 10,
> I've used the JDK from [1] and works like a charm.
>
> Cheers,
> Michael
>
> [1]
> http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
>
> --
> Michael Hausenblas
> Ireland, Europe
> http://mhausenblas.info/
>
> On 14 Jan 2013, at 16:37, Jacques Nadeau <ja...@gmail.com> wrote:
>
> > Given that Java6 is public updates EOL next month, I've targeted Java7.
> > You probably need to update your JDK.
> >
> > Jacques
> >
> > On Mon, Jan 14, 2013 at 4:31 PM, Michael Hausenblas <
> > michael.hausenblas@gmail.com> wrote:
> >
> >>
> >>> Be careful, there are many sharp edges.
> >>>
> >>> :)
> >>
> >>
> >> Alright ;)
> >>
> >> So here we go: 'mvn validate' works fine for me but then ...
> >>
> >> $ mvn verify
> >>
> >> [INFO] prototype-parent .................................. SUCCESS
> [7.166s]
> >> [INFO] common ............................................ FAILURE
> >> [17.775s]
> >> [INFO] contrib-parent .................................... SKIPPED
> >> [INFO] pom ............................................... SKIPPED
> >> [INFO] exec .............................................. SKIPPED
> >> [INFO] java-exec ......................................... SKIPPED
> >> [INFO] Logical Plan Execution Reference Implementation ... SKIPPED
> >> [INFO] planner ........................................... SKIPPED
> >> [INFO] sqlparser ......................................... SKIPPED
> >>
> >>
> >> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> execute
> >> goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
> >> (default-compile) on project common: Compilation failure
> >> Failure executing javac, but could not parse the error:
> >> javac: invalid target release: 1.7
> >> Usage: javac <options> <source files>
> >> use -help for a list of possible options
> >>
> >>
> >> at
> >>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
> >> at
> >>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> >> at
> >>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> >> at
> >>
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> >> at
> >>
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> >> at
> >>
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> >> at
> >>
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> >> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
> >> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> >> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> >> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> >> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at
> >>
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> >> at
> >>
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> >> at
> >>
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> >> at
> >>
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> >> Caused by: org.apache.maven.plugin.CompilationFailureException:
> >> Compilation failure
> >> Failure executing javac, but could not parse the error:
> >> javac: invalid target release: 1.7
> >> Usage: javac <options> <source files>
> >> use -help for a list of possible options
> >>
> >>
> >> at
> >>
> org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
> >> at
> >> org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114)
> >> at
> >>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> >> at
> >>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> >>
> >>
> >>
> >> Cheers,
> >> Michael
> >>
> >> --
> >> Michael Hausenblas
> >> Ireland, Europe
> >> http://mhausenblas.info/
> >>
> >> On 14 Jan 2013, at 16:06, Jacques Nadeau <ja...@gmail.com>
> wrote:
> >>
> >>> Be careful, there are many sharp edges.
> >>>
> >>> :)
> >>>
> >>> On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas <
> >>> michael.hausenblas@gmail.com> wrote:
> >>>
> >>>>
> >>>> Cool stuff, Jacques - will give it a shot ASAP!
> >>>>
> >>>> Cheers,
> >>>> Michael
> >>>>
> >>>> --
> >>>> Michael Hausenblas
> >>>> Ireland, Europe
> >>>> http://mhausenblas.info/
> >>>>
> >>>> On 14 Jan 2013, at 15:56, Jacques Nadeau <ja...@gmail.com>
> >> wrote:
> >>>>
> >>>>> I've been pulling together a reference logical plan interpreter. I'm
> >>>>> working with Ted to get it inside the Drill sandbox. For now, you
> can
> >>>> find
> >>>>> it on my repo at
> https://github.com/jacques-n/incubator-drill(prototype
> >>>>> branch)
> >>>>>
> >>>>>
> >>>>>
> >>>>> The goals of the reference interpreter are:
> >>>>>
> >>>>>
> >>>>> - To provide a simple way to run a Logical Plan against some sample
> >>>> data
> >>>>> and get back the expected result
> >>>>> - Allow work to start on the parsers while we scale up the
> performance
> >>>>> and capabilities of the execution engine and optimizer.
> >>>>> - Allow evaluation work on particular technical approaches such as
> >>>>> exploring the impact of hierarchical and schema less data on query
> >>>>> evaluation.
> >>>>>
> >>>>> These goals do not include performance, memory handling, or
> >>>>> efficiency. Currently,
> >>>>> the interpreter is a single node/thread process. This will change
> >>>> shortly
> >>>>> so that it also run as a clustered process.
> >>>>>
> >>>>> The entry point is inside the /sandbox/prototype/exec/ref module:
> >>>>> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example
> >>>> program
> >>>>> utilizes two resources: simple-plan.json and donuts.json and outputs
> >> data
> >>>>> to /opt/data/out.json.
> >>>>>
> >>>>>
> >>>>> Some of things that 'work'.
> >>>>>
> >>>>>
> >>>>> - Read/write basic json.
> >>>>> - ROPs (reference operators): Filter, Transform, Group, Aggregate
> >>>>> (simple), Order, Union.
> >>>>> - Example aggregate and basic functions including sum, count,
> >> multiply,
> >>>>> add, compare, equals.
> >>>>>
> >>>>> Basic glossary/concepts (we'll get this on the wiki/javadocs):
> >>>>>
> >>>>>
> >>>>> - LOP: Logical Operator. An implementation agnostic data flow
> >> operator
> >>>>> utilized by the Logical Plan.
> >>>>> - ROP: Reference Operator: A reference operator implementation that
> >>>>> pairs with a LOP.
> >>>>> - FunctionDefinition: A definition of a particular function.
> >> Describes
> >>>>> a set of aliases, an allowable set of input arguments and an
> interface
> >>>> that
> >>>>> will attempt to determine output type.
> >>>>> - BasicEvaluator: An implementation of a particular non-aggregate
> >>>>> expression. Receives a record pointer at creation time. Returns a
> >>>>> DataValue.
> >>>>> - AggregateEvaluator: An implementation of a particular aggregating
> >>>>> function. Is provided a record pointer at creation time. Expects
> >>>> regular
> >>>>> calls to addRecord() followed by a call to eval() which provides the
> >>>>> aggregate value.
> >>>>> - DataValue: A pointer to a particular data value. Implementation
> >>>>> classes includes things like ScalarLong, ScalarBytes, SimpleMapValue
> >>>> and
> >>>>> SimpleArrayValue.
> >>>>>
> >>>>> The standard record iterator utilized between each ROP utilizes the
> >>>>> org.apache.drill.exec.ref.RecordIterator interface. This is somewhat
> >>>>> inspired by the AttributeSource concepts from within the Lucene
> >> project.
> >>>>> (I'm planning to extend these concepts all the way to the individual
> >>>>> DataValues.)
> >>>>>
> >>>>>
> >>>>>
> >>>>> My next goals are to add tests, finish adding ROPs, add local and
> >> remote
> >>>>> exchange nodes (parallelization), add a bunch of documentation and
> >>>> extract
> >>>>> out the Execution plan as a separate intermediate representation.
> >>>>>
> >>>>>
> >>>>>
> >>>>> It needs a lot more evaluators to be a true reference interpreter (as
> >>>> well
> >>>>> as the rest of the ROPs). The existing ones can be utilized as
> >>>> prototypes.
> >>>>> Anyone interested in ripping through a bunch of additional evaluators
> >> and
> >>>>> associated FunctionDefinitions?
> >>>>
> >>>>
> >>
> >>
>
>
Re: First pass at a reference interpreter
Posted by Michael Hausenblas <mi...@gmail.com>.
> Given that Java6 is public updates EOL next month, I've targeted Java7.
> You probably need to update your JDK.
Good point. I suppose this should be in big red blinking letters on top of the installation instructions.
'mvn verify' is a success now for me ... in case anyone is on MacOS 10, I've used the JDK from [1] and works like a charm.
Cheers,
Michael
[1] http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/
On 14 Jan 2013, at 16:37, Jacques Nadeau <ja...@gmail.com> wrote:
> Given that Java6 is public updates EOL next month, I've targeted Java7.
> You probably need to update your JDK.
>
> Jacques
>
> On Mon, Jan 14, 2013 at 4:31 PM, Michael Hausenblas <
> michael.hausenblas@gmail.com> wrote:
>
>>
>>> Be careful, there are many sharp edges.
>>>
>>> :)
>>
>>
>> Alright ;)
>>
>> So here we go: 'mvn validate' works fine for me but then ...
>>
>> $ mvn verify
>>
>> [INFO] prototype-parent .................................. SUCCESS [7.166s]
>> [INFO] common ............................................ FAILURE
>> [17.775s]
>> [INFO] contrib-parent .................................... SKIPPED
>> [INFO] pom ............................................... SKIPPED
>> [INFO] exec .............................................. SKIPPED
>> [INFO] java-exec ......................................... SKIPPED
>> [INFO] Logical Plan Execution Reference Implementation ... SKIPPED
>> [INFO] planner ........................................... SKIPPED
>> [INFO] sqlparser ......................................... SKIPPED
>>
>>
>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
>> goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
>> (default-compile) on project common: Compilation failure
>> Failure executing javac, but could not parse the error:
>> javac: invalid target release: 1.7
>> Usage: javac <options> <source files>
>> use -help for a list of possible options
>>
>>
>> at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
>> at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>> at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>> at
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
>> at
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
>> at
>> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
>> at
>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
>> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
>> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
>> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
>> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
>> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
>> Caused by: org.apache.maven.plugin.CompilationFailureException:
>> Compilation failure
>> Failure executing javac, but could not parse the error:
>> javac: invalid target release: 1.7
>> Usage: javac <options> <source files>
>> use -help for a list of possible options
>>
>>
>> at
>> org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
>> at
>> org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114)
>> at
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>> at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
>>
>>
>>
>> Cheers,
>> Michael
>>
>> --
>> Michael Hausenblas
>> Ireland, Europe
>> http://mhausenblas.info/
>>
>> On 14 Jan 2013, at 16:06, Jacques Nadeau <ja...@gmail.com> wrote:
>>
>>> Be careful, there are many sharp edges.
>>>
>>> :)
>>>
>>> On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas <
>>> michael.hausenblas@gmail.com> wrote:
>>>
>>>>
>>>> Cool stuff, Jacques - will give it a shot ASAP!
>>>>
>>>> Cheers,
>>>> Michael
>>>>
>>>> --
>>>> Michael Hausenblas
>>>> Ireland, Europe
>>>> http://mhausenblas.info/
>>>>
>>>> On 14 Jan 2013, at 15:56, Jacques Nadeau <ja...@gmail.com>
>> wrote:
>>>>
>>>>> I've been pulling together a reference logical plan interpreter. I'm
>>>>> working with Ted to get it inside the Drill sandbox. For now, you can
>>>> find
>>>>> it on my repo at https://github.com/jacques-n/incubator-drill(prototype
>>>>> branch)
>>>>>
>>>>>
>>>>>
>>>>> The goals of the reference interpreter are:
>>>>>
>>>>>
>>>>> - To provide a simple way to run a Logical Plan against some sample
>>>> data
>>>>> and get back the expected result
>>>>> - Allow work to start on the parsers while we scale up the performance
>>>>> and capabilities of the execution engine and optimizer.
>>>>> - Allow evaluation work on particular technical approaches such as
>>>>> exploring the impact of hierarchical and schema less data on query
>>>>> evaluation.
>>>>>
>>>>> These goals do not include performance, memory handling, or
>>>>> efficiency. Currently,
>>>>> the interpreter is a single node/thread process. This will change
>>>> shortly
>>>>> so that it also run as a clustered process.
>>>>>
>>>>> The entry point is inside the /sandbox/prototype/exec/ref module:
>>>>> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example
>>>> program
>>>>> utilizes two resources: simple-plan.json and donuts.json and outputs
>> data
>>>>> to /opt/data/out.json.
>>>>>
>>>>>
>>>>> Some of things that 'work'.
>>>>>
>>>>>
>>>>> - Read/write basic json.
>>>>> - ROPs (reference operators): Filter, Transform, Group, Aggregate
>>>>> (simple), Order, Union.
>>>>> - Example aggregate and basic functions including sum, count,
>> multiply,
>>>>> add, compare, equals.
>>>>>
>>>>> Basic glossary/concepts (we'll get this on the wiki/javadocs):
>>>>>
>>>>>
>>>>> - LOP: Logical Operator. An implementation agnostic data flow
>> operator
>>>>> utilized by the Logical Plan.
>>>>> - ROP: Reference Operator: A reference operator implementation that
>>>>> pairs with a LOP.
>>>>> - FunctionDefinition: A definition of a particular function.
>> Describes
>>>>> a set of aliases, an allowable set of input arguments and an interface
>>>> that
>>>>> will attempt to determine output type.
>>>>> - BasicEvaluator: An implementation of a particular non-aggregate
>>>>> expression. Receives a record pointer at creation time. Returns a
>>>>> DataValue.
>>>>> - AggregateEvaluator: An implementation of a particular aggregating
>>>>> function. Is provided a record pointer at creation time. Expects
>>>> regular
>>>>> calls to addRecord() followed by a call to eval() which provides the
>>>>> aggregate value.
>>>>> - DataValue: A pointer to a particular data value. Implementation
>>>>> classes includes things like ScalarLong, ScalarBytes, SimpleMapValue
>>>> and
>>>>> SimpleArrayValue.
>>>>>
>>>>> The standard record iterator utilized between each ROP utilizes the
>>>>> org.apache.drill.exec.ref.RecordIterator interface. This is somewhat
>>>>> inspired by the AttributeSource concepts from within the Lucene
>> project.
>>>>> (I'm planning to extend these concepts all the way to the individual
>>>>> DataValues.)
>>>>>
>>>>>
>>>>>
>>>>> My next goals are to add tests, finish adding ROPs, add local and
>> remote
>>>>> exchange nodes (parallelization), add a bunch of documentation and
>>>> extract
>>>>> out the Execution plan as a separate intermediate representation.
>>>>>
>>>>>
>>>>>
>>>>> It needs a lot more evaluators to be a true reference interpreter (as
>>>> well
>>>>> as the rest of the ROPs). The existing ones can be utilized as
>>>> prototypes.
>>>>> Anyone interested in ripping through a bunch of additional evaluators
>> and
>>>>> associated FunctionDefinitions?
>>>>
>>>>
>>
>>
Re: First pass at a reference interpreter
Posted by Jacques Nadeau <ja...@gmail.com>.
Given that Java6 is public updates EOL next month, I've targeted Java7.
You probably need to update your JDK.
Jacques
On Mon, Jan 14, 2013 at 4:31 PM, Michael Hausenblas <
michael.hausenblas@gmail.com> wrote:
>
> > Be careful, there are many sharp edges.
> >
> > :)
>
>
> Alright ;)
>
> So here we go: 'mvn validate' works fine for me but then ...
>
> $ mvn verify
>
> [INFO] prototype-parent .................................. SUCCESS [7.166s]
> [INFO] common ............................................ FAILURE
> [17.775s]
> [INFO] contrib-parent .................................... SKIPPED
> [INFO] pom ............................................... SKIPPED
> [INFO] exec .............................................. SKIPPED
> [INFO] java-exec ......................................... SKIPPED
> [INFO] Logical Plan Execution Reference Implementation ... SKIPPED
> [INFO] planner ........................................... SKIPPED
> [INFO] sqlparser ......................................... SKIPPED
>
>
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
> (default-compile) on project common: Compilation failure
> Failure executing javac, but could not parse the error:
> javac: invalid target release: 1.7
> Usage: javac <options> <source files>
> use -help for a list of possible options
>
>
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> Caused by: org.apache.maven.plugin.CompilationFailureException:
> Compilation failure
> Failure executing javac, but could not parse the error:
> javac: invalid target release: 1.7
> Usage: javac <options> <source files>
> use -help for a list of possible options
>
>
> at
> org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
> at
> org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
>
>
>
> Cheers,
> Michael
>
> --
> Michael Hausenblas
> Ireland, Europe
> http://mhausenblas.info/
>
> On 14 Jan 2013, at 16:06, Jacques Nadeau <ja...@gmail.com> wrote:
>
> > Be careful, there are many sharp edges.
> >
> > :)
> >
> > On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas <
> > michael.hausenblas@gmail.com> wrote:
> >
> >>
> >> Cool stuff, Jacques - will give it a shot ASAP!
> >>
> >> Cheers,
> >> Michael
> >>
> >> --
> >> Michael Hausenblas
> >> Ireland, Europe
> >> http://mhausenblas.info/
> >>
> >> On 14 Jan 2013, at 15:56, Jacques Nadeau <ja...@gmail.com>
> wrote:
> >>
> >>> I've been pulling together a reference logical plan interpreter. I'm
> >>> working with Ted to get it inside the Drill sandbox. For now, you can
> >> find
> >>> it on my repo at https://github.com/jacques-n/incubator-drill(prototype
> >>> branch)
> >>>
> >>>
> >>>
> >>> The goals of the reference interpreter are:
> >>>
> >>>
> >>> - To provide a simple way to run a Logical Plan against some sample
> >> data
> >>> and get back the expected result
> >>> - Allow work to start on the parsers while we scale up the performance
> >>> and capabilities of the execution engine and optimizer.
> >>> - Allow evaluation work on particular technical approaches such as
> >>> exploring the impact of hierarchical and schema less data on query
> >>> evaluation.
> >>>
> >>> These goals do not include performance, memory handling, or
> >>> efficiency. Currently,
> >>> the interpreter is a single node/thread process. This will change
> >> shortly
> >>> so that it also run as a clustered process.
> >>>
> >>> The entry point is inside the /sandbox/prototype/exec/ref module:
> >>> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example
> >> program
> >>> utilizes two resources: simple-plan.json and donuts.json and outputs
> data
> >>> to /opt/data/out.json.
> >>>
> >>>
> >>> Some of things that 'work'.
> >>>
> >>>
> >>> - Read/write basic json.
> >>> - ROPs (reference operators): Filter, Transform, Group, Aggregate
> >>> (simple), Order, Union.
> >>> - Example aggregate and basic functions including sum, count,
> multiply,
> >>> add, compare, equals.
> >>>
> >>> Basic glossary/concepts (we'll get this on the wiki/javadocs):
> >>>
> >>>
> >>> - LOP: Logical Operator. An implementation agnostic data flow
> operator
> >>> utilized by the Logical Plan.
> >>> - ROP: Reference Operator: A reference operator implementation that
> >>> pairs with a LOP.
> >>> - FunctionDefinition: A definition of a particular function.
> Describes
> >>> a set of aliases, an allowable set of input arguments and an interface
> >> that
> >>> will attempt to determine output type.
> >>> - BasicEvaluator: An implementation of a particular non-aggregate
> >>> expression. Receives a record pointer at creation time. Returns a
> >>> DataValue.
> >>> - AggregateEvaluator: An implementation of a particular aggregating
> >>> function. Is provided a record pointer at creation time. Expects
> >> regular
> >>> calls to addRecord() followed by a call to eval() which provides the
> >>> aggregate value.
> >>> - DataValue: A pointer to a particular data value. Implementation
> >>> classes includes things like ScalarLong, ScalarBytes, SimpleMapValue
> >> and
> >>> SimpleArrayValue.
> >>>
> >>> The standard record iterator utilized between each ROP utilizes the
> >>> org.apache.drill.exec.ref.RecordIterator interface. This is somewhat
> >>> inspired by the AttributeSource concepts from within the Lucene
> project.
> >>> (I'm planning to extend these concepts all the way to the individual
> >>> DataValues.)
> >>>
> >>>
> >>>
> >>> My next goals are to add tests, finish adding ROPs, add local and
> remote
> >>> exchange nodes (parallelization), add a bunch of documentation and
> >> extract
> >>> out the Execution plan as a separate intermediate representation.
> >>>
> >>>
> >>>
> >>> It needs a lot more evaluators to be a true reference interpreter (as
> >> well
> >>> as the rest of the ROPs). The existing ones can be utilized as
> >> prototypes.
> >>> Anyone interested in ripping through a bunch of additional evaluators
> and
> >>> associated FunctionDefinitions?
> >>
> >>
>
>
Re: First pass at a reference interpreter
Posted by Michael Hausenblas <mi...@gmail.com>.
> Be careful, there are many sharp edges.
>
> :)
Alright ;)
So here we go: 'mvn validate' works fine for me but then ...
$ mvn verify
[INFO] prototype-parent .................................. SUCCESS [7.166s]
[INFO] common ............................................ FAILURE [17.775s]
[INFO] contrib-parent .................................... SKIPPED
[INFO] pom ............................................... SKIPPED
[INFO] exec .............................................. SKIPPED
[INFO] java-exec ......................................... SKIPPED
[INFO] Logical Plan Execution Reference Implementation ... SKIPPED
[INFO] planner ........................................... SKIPPED
[INFO] sqlparser ......................................... SKIPPED
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project common: Compilation failure
Failure executing javac, but could not parse the error:
javac: invalid target release: 1.7
Usage: javac <options> <source files>
use -help for a list of possible options
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure
Failure executing javac, but could not parse the error:
javac: invalid target release: 1.7
Usage: javac <options> <source files>
use -help for a list of possible options
at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
at org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
Cheers,
Michael
--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/
On 14 Jan 2013, at 16:06, Jacques Nadeau <ja...@gmail.com> wrote:
> Be careful, there are many sharp edges.
>
> :)
>
> On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas <
> michael.hausenblas@gmail.com> wrote:
>
>>
>> Cool stuff, Jacques - will give it a shot ASAP!
>>
>> Cheers,
>> Michael
>>
>> --
>> Michael Hausenblas
>> Ireland, Europe
>> http://mhausenblas.info/
>>
>> On 14 Jan 2013, at 15:56, Jacques Nadeau <ja...@gmail.com> wrote:
>>
>>> I've been pulling together a reference logical plan interpreter. I'm
>>> working with Ted to get it inside the Drill sandbox. For now, you can
>> find
>>> it on my repo at https://github.com/jacques-n/incubator-drill (prototype
>>> branch)
>>>
>>>
>>>
>>> The goals of the reference interpreter are:
>>>
>>>
>>> - To provide a simple way to run a Logical Plan against some sample
>> data
>>> and get back the expected result
>>> - Allow work to start on the parsers while we scale up the performance
>>> and capabilities of the execution engine and optimizer.
>>> - Allow evaluation work on particular technical approaches such as
>>> exploring the impact of hierarchical and schema less data on query
>>> evaluation.
>>>
>>> These goals do not include performance, memory handling, or
>>> efficiency. Currently,
>>> the interpreter is a single node/thread process. This will change
>> shortly
>>> so that it also run as a clustered process.
>>>
>>> The entry point is inside the /sandbox/prototype/exec/ref module:
>>> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example
>> program
>>> utilizes two resources: simple-plan.json and donuts.json and outputs data
>>> to /opt/data/out.json.
>>>
>>>
>>> Some of things that 'work'.
>>>
>>>
>>> - Read/write basic json.
>>> - ROPs (reference operators): Filter, Transform, Group, Aggregate
>>> (simple), Order, Union.
>>> - Example aggregate and basic functions including sum, count, multiply,
>>> add, compare, equals.
>>>
>>> Basic glossary/concepts (we'll get this on the wiki/javadocs):
>>>
>>>
>>> - LOP: Logical Operator. An implementation agnostic data flow operator
>>> utilized by the Logical Plan.
>>> - ROP: Reference Operator: A reference operator implementation that
>>> pairs with a LOP.
>>> - FunctionDefinition: A definition of a particular function. Describes
>>> a set of aliases, an allowable set of input arguments and an interface
>> that
>>> will attempt to determine output type.
>>> - BasicEvaluator: An implementation of a particular non-aggregate
>>> expression. Receives a record pointer at creation time. Returns a
>>> DataValue.
>>> - AggregateEvaluator: An implementation of a particular aggregating
>>> function. Is provided a record pointer at creation time. Expects
>> regular
>>> calls to addRecord() followed by a call to eval() which provides the
>>> aggregate value.
>>> - DataValue: A pointer to a particular data value. Implementation
>>> classes includes things like ScalarLong, ScalarBytes, SimpleMapValue
>> and
>>> SimpleArrayValue.
>>>
>>> The standard record iterator utilized between each ROP utilizes the
>>> org.apache.drill.exec.ref.RecordIterator interface. This is somewhat
>>> inspired by the AttributeSource concepts from within the Lucene project.
>>> (I'm planning to extend these concepts all the way to the individual
>>> DataValues.)
>>>
>>>
>>>
>>> My next goals are to add tests, finish adding ROPs, add local and remote
>>> exchange nodes (parallelization), add a bunch of documentation and
>> extract
>>> out the Execution plan as a separate intermediate representation.
>>>
>>>
>>>
>>> It needs a lot more evaluators to be a true reference interpreter (as
>> well
>>> as the rest of the ROPs). The existing ones can be utilized as
>> prototypes.
>>> Anyone interested in ripping through a bunch of additional evaluators and
>>> associated FunctionDefinitions?
>>
>>
Re: First pass at a reference interpreter
Posted by Jacques Nadeau <ja...@gmail.com>.
Be careful, there are many sharp edges.
:)
On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas <
michael.hausenblas@gmail.com> wrote:
>
> Cool stuff, Jacques - will give it a shot ASAP!
>
> Cheers,
> Michael
>
> --
> Michael Hausenblas
> Ireland, Europe
> http://mhausenblas.info/
>
> On 14 Jan 2013, at 15:56, Jacques Nadeau <ja...@gmail.com> wrote:
>
> > I've been pulling together a reference logical plan interpreter. I'm
> > working with Ted to get it inside the Drill sandbox. For now, you can
> find
> > it on my repo at https://github.com/jacques-n/incubator-drill (prototype
> > branch)
> >
> >
> >
> > The goals of the reference interpreter are:
> >
> >
> > - To provide a simple way to run a Logical Plan against some sample
> data
> > and get back the expected result
> > - Allow work to start on the parsers while we scale up the performance
> > and capabilities of the execution engine and optimizer.
> > - Allow evaluation work on particular technical approaches such as
> > exploring the impact of hierarchical and schema less data on query
> > evaluation.
> >
> > These goals do not include performance, memory handling, or
> > efficiency. Currently,
> > the interpreter is a single node/thread process. This will change
> shortly
> > so that it also run as a clustered process.
> >
> > The entry point is inside the /sandbox/prototype/exec/ref module:
> > org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example
> program
> > utilizes two resources: simple-plan.json and donuts.json and outputs data
> > to /opt/data/out.json.
> >
> >
> > Some of things that 'work'.
> >
> >
> > - Read/write basic json.
> > - ROPs (reference operators): Filter, Transform, Group, Aggregate
> > (simple), Order, Union.
> > - Example aggregate and basic functions including sum, count, multiply,
> > add, compare, equals.
> >
> > Basic glossary/concepts (we'll get this on the wiki/javadocs):
> >
> >
> > - LOP: Logical Operator. An implementation agnostic data flow operator
> > utilized by the Logical Plan.
> > - ROP: Reference Operator: A reference operator implementation that
> > pairs with a LOP.
> > - FunctionDefinition: A definition of a particular function. Describes
> > a set of aliases, an allowable set of input arguments and an interface
> that
> > will attempt to determine output type.
> > - BasicEvaluator: An implementation of a particular non-aggregate
> > expression. Receives a record pointer at creation time. Returns a
> > DataValue.
> > - AggregateEvaluator: An implementation of a particular aggregating
> > function. Is provided a record pointer at creation time. Expects
> regular
> > calls to addRecord() followed by a call to eval() which provides the
> > aggregate value.
> > - DataValue: A pointer to a particular data value. Implementation
> > classes includes things like ScalarLong, ScalarBytes, SimpleMapValue
> and
> > SimpleArrayValue.
> >
> > The standard record iterator utilized between each ROP utilizes the
> > org.apache.drill.exec.ref.RecordIterator interface. This is somewhat
> > inspired by the AttributeSource concepts from within the Lucene project.
> > (I'm planning to extend these concepts all the way to the individual
> > DataValues.)
> >
> >
> >
> > My next goals are to add tests, finish adding ROPs, add local and remote
> > exchange nodes (parallelization), add a bunch of documentation and
> extract
> > out the Execution plan as a separate intermediate representation.
> >
> >
> >
> > It needs a lot more evaluators to be a true reference interpreter (as
> well
> > as the rest of the ROPs). The existing ones can be utilized as
> prototypes.
> > Anyone interested in ripping through a bunch of additional evaluators and
> > associated FunctionDefinitions?
>
>
Re: First pass at a reference interpreter
Posted by Michael Hausenblas <mi...@gmail.com>.
Cool stuff, Jacques - will give it a shot ASAP!
Cheers,
Michael
--
Michael Hausenblas
Ireland, Europe
http://mhausenblas.info/
On 14 Jan 2013, at 15:56, Jacques Nadeau <ja...@gmail.com> wrote:
> I've been pulling together a reference logical plan interpreter. I'm
> working with Ted to get it inside the Drill sandbox. For now, you can find
> it on my repo at https://github.com/jacques-n/incubator-drill (prototype
> branch)
>
>
>
> The goals of the reference interpreter are:
>
>
> - To provide a simple way to run a Logical Plan against some sample data
> and get back the expected result
> - Allow work to start on the parsers while we scale up the performance
> and capabilities of the execution engine and optimizer.
> - Allow evaluation work on particular technical approaches such as
> exploring the impact of hierarchical and schema less data on query
> evaluation.
>
> These goals do not include performance, memory handling, or
> efficiency. Currently,
> the interpreter is a single node/thread process. This will change shortly
> so that it also run as a clustered process.
>
> The entry point is inside the /sandbox/prototype/exec/ref module:
> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example program
> utilizes two resources: simple-plan.json and donuts.json and outputs data
> to /opt/data/out.json.
>
>
> Some of things that 'work'.
>
>
> - Read/write basic json.
> - ROPs (reference operators): Filter, Transform, Group, Aggregate
> (simple), Order, Union.
> - Example aggregate and basic functions including sum, count, multiply,
> add, compare, equals.
>
> Basic glossary/concepts (we'll get this on the wiki/javadocs):
>
>
> - LOP: Logical Operator. An implementation agnostic data flow operator
> utilized by the Logical Plan.
> - ROP: Reference Operator: A reference operator implementation that
> pairs with a LOP.
> - FunctionDefinition: A definition of a particular function. Describes
> a set of aliases, an allowable set of input arguments and an interface that
> will attempt to determine output type.
> - BasicEvaluator: An implementation of a particular non-aggregate
> expression. Receives a record pointer at creation time. Returns a
> DataValue.
> - AggregateEvaluator: An implementation of a particular aggregating
> function. Is provided a record pointer at creation time. Expects regular
> calls to addRecord() followed by a call to eval() which provides the
> aggregate value.
> - DataValue: A pointer to a particular data value. Implementation
> classes includes things like ScalarLong, ScalarBytes, SimpleMapValue and
> SimpleArrayValue.
>
> The standard record iterator utilized between each ROP utilizes the
> org.apache.drill.exec.ref.RecordIterator interface. This is somewhat
> inspired by the AttributeSource concepts from within the Lucene project.
> (I'm planning to extend these concepts all the way to the individual
> DataValues.)
>
>
>
> My next goals are to add tests, finish adding ROPs, add local and remote
> exchange nodes (parallelization), add a bunch of documentation and extract
> out the Execution plan as a separate intermediate representation.
>
>
>
> It needs a lot more evaluators to be a true reference interpreter (as well
> as the rest of the ROPs). The existing ones can be utilized as prototypes.
> Anyone interested in ripping through a bunch of additional evaluators and
> associated FunctionDefinitions?