purpose of using linq4j as an intermediate representation

["Hartman, Trevor" <th...@ebay.com>]

I've modeled my Calcite adapter after the Mongo adapter, and I'm wondering what the purpose is of representing the call to MongoMethod.MONGO_QUERYABLE_AGGREGATE.method as a linq4j expression here [1]? I'm having a hard time encoding the projection data I need to get into my Enumerator, particularly since I'm using Scala. It seems like this is a really hard roundabout way to essentially create a lambda. Or are there reasons besides delayed execution for using linq4j? Does some part of Calcite need to be able to walk the expression tree?

Thanks,
Trevor

[1] https://github.com/apache/incubator-calcite/blob/master/mongodb%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fcalcite%2Fadapter%2Fmongodb%2FMongoToEnumerableConverter.java#L100-L137

Re: purpose of using linq4j as an intermediate representation

["Hartman, Trevor" <th...@ebay.com>]

FYI Apache Drill is a query engine built on top of Calcite that supports
hierarchical data and columnar storage using Parquet. Depending on where
your development is, you may want to take a look at it.

Thanks for the heads up. My coworker looked at Drill before building our own store. I wasn't involved in the project at the time so I don't know the specifics, but for some reason it didn't meet our requirements. At this point we've been using our system in production for about 7 months with a not-very-friendly Scala-based query interface, which is why I'm working on adding Calcite. I'll check out Drill's source to see how you guys are using Calcite.

Re: purpose of using linq4j as an intermediate representation

[Jacques Nadeau <ja...@apache.org>]

>I'm working on a columnar data store that supports hierarchical schemas

FYI Apache Drill is a query engine built on top of Calcite that supports
hierarchical data and columnar storage using Parquet.  Depending on where
your development is, you may want to take a look at it.

On Fri, Dec 12, 2014 at 12:46 PM, Hartman, Trevor <th...@ebay.com> wrote:

> I'm working on a columnar data store that supports hierarchical schemas.
> It's not open source (yet), so I've created a demo repo that mirrors
> exactly what I'm trying to do.
>
> In MapProjectRel, I get some data from `getNamedProjects` and store it on
> the implementor [1]. This data structure (Seq[String]) is not sufficient to
> capture nested projections, but I haven't fixed that yet because my main
> problem is the linq4j translation like I mentioned, which I documented in
> MapToEnumerableConverter [2]. Even getting the raw List[RexNode] from
> `getProjections` would have the data required for my MapEnumerator to get
> the right fields while it scans the dataset.
>
> I don't know whether it makes more sense to use the Enumerable or
> Interpreter calling convention. Performance does matter here as this will
> hopefully become the primary way in which live apps will get data out of
> the system. Enumerable sounds nice in that I can over time pushdown each
> piece as I  get to it, and still have Calcite provide the other operation
> in the meantime. Would Interpreter be able to do the same?
>
> What I've been trying to do is get all the necessary data structures
> containing projections, sorting and aggregation info into my MapEnumerator,
> because that's where the query interface to the underlying database lives
> in my actual project.
>
> Thanks,
> Trevor
>
> [1]
> https://github.com/devth/calcite-map-demo/blob/master/src%2Fmain%2Fscala%2Fdevth%2Fcalcite%2FMapProjectRel.scala#L36
>
> [2]
> https://github.com/devth/calcite-map-demo/blob/master/src%2Fmain%2Fscala%2Fdevth%2Fcalcite%2FMapToEnumerableConverter.scala#L45-L55
>

Re: purpose of using linq4j as an intermediate representation

[Vladimir Sitnikov <si...@gmail.com>]

Good you got it working.

It looks like you were bitten by "PhysType" issue a bit (Object[] vs Map).
I hope this would be simplified as well in the near future.
Vladimir

Re: purpose of using linq4j as an intermediate representation

["Hartman, Trevor" <th...@ebay.com>]

Vladimir, thanks for fixing CALCITE-506! I was able to use the new .stash to pass my List[RexNode] projections through to the enumerator [1].

Trevor

[1] https://github.com/devth/calcite-map-demo/blob/master/src/main/scala/devth/calcite/MapToEnumerableConverter.scala#L44-49

Re: purpose of using linq4j as an intermediate representation

["Hartman, Trevor" <th...@ebay.com>]

Oops, confused RelNode and RexNode. Looks like I would need CALCITE-506 after all before taking this approach.


On December 12, 2014 at 3:45:48 PM, Hartman, Trevor (thartman@ebay.com<ma...@ebay.com>) wrote:

I understand updating the signature, but no idea what the implementation would look like. This sounds like a good approach to my problem though. And since current stash accepts RelNode, I might be able to use it as-is, though I don't understand the purpose of the second parameter, Class clazz.

Trevor

Re: purpose of using linq4j as an intermediate representation

["Hartman, Trevor" <th...@ebay.com>]

I understand updating the signature, but no idea what the implementation would look like. This sounds like a good approach to my problem though. And since current stash accepts RelNode, I might be able to use it as-is, though I don't understand the purpose of the second parameter, Class clazz.

Trevor

Re: purpose of using linq4j as an intermediate representation

[Vladimir Sitnikov <si...@gmail.com>]

> What I've been trying to do is get all the necessary data structures containing projections, sorting and aggregation info into my MapEnumerator

Did you consider RexFieldAccess as an intermediate representation?
E.g. just reuse the expression from Project as is.

> I haven't fixed that yet because my main problem is the linq4j translation like I mentioned

Suppose you use RexFieldAccess as the representation. Then you'll have
to pass Seq[RexFieldAccess].
Here's where `.stash` applies: you call
enumerableImplementor.stash(yourSeqOfFields) and it gives you the
linq4j "Expression" that you use as "fieldsExpression".

The idea behind this EnumerableRelImplementor.stash is to pass
arbitrary object without involving complex linq4j code.

If that sounds good, you can try to update `stash` signature as in CALCITE-506.

-- 
Vladimir

Re: purpose of using linq4j as an intermediate representation

["Hartman, Trevor" <th...@ebay.com>]

I'm working on a columnar data store that supports hierarchical schemas. It's not open source (yet), so I've created a demo repo that mirrors exactly what I'm trying to do.

In MapProjectRel, I get some data from `getNamedProjects` and store it on the implementor [1]. This data structure (Seq[String]) is not sufficient to capture nested projections, but I haven't fixed that yet because my main problem is the linq4j translation like I mentioned, which I documented in MapToEnumerableConverter [2]. Even getting the raw List[RexNode] from `getProjections` would have the data required for my MapEnumerator to get the right fields while it scans the dataset.

I don't know whether it makes more sense to use the Enumerable or Interpreter calling convention. Performance does matter here as this will hopefully become the primary way in which live apps will get data out of the system. Enumerable sounds nice in that I can over time pushdown each piece as I  get to it, and still have Calcite provide the other operation in the meantime. Would Interpreter be able to do the same?

What I've been trying to do is get all the necessary data structures containing projections, sorting and aggregation info into my MapEnumerator, because that's where the query interface to the underlying database lives in my actual project.

Thanks,
Trevor

[1] https://github.com/devth/calcite-map-demo/blob/master/src%2Fmain%2Fscala%2Fdevth%2Fcalcite%2FMapProjectRel.scala#L36

[2] https://github.com/devth/calcite-map-demo/blob/master/src%2Fmain%2Fscala%2Fdevth%2Fcalcite%2FMapToEnumerableConverter.scala#L45-L55

Re: purpose of using linq4j as an intermediate representation

[Julian Hyde <ju...@gmail.com>]

Vladimir is correct but I am working hard to display Enumerable convention’s status as “built-in”.

The data has to come out of Mongo and into JDBC somehow, so the plan needs to bridge into a calling convention that Calcite understands. Until recently Enumerable has been the calling convention of choice. It involves translating the tree of RelNodes into java source code that implement the Enumerable interface (similar to Iterable).

In the case of Mongo this was a thin layer — execute a Mongo query using DBCollection.aggregate, and convert to an Enumerable — but it had to exist because Enumerable was the only convention.

Now we have Interpreter and interfaces such as ScannableTable a better way, especially for gateways that are not performance-critical, would be to just return an interpretable RelNode. That will take a little work (we need to make the translation to interpreter nodes a bit more extensible) but not much.

Trevor, can you share what you are working on? I could work with you to get it into interpreter convention, if you like.

Also, the comment thread on https://issues.apache.org/jira/browse/CALCITE-506 covers similar territory: “I'd love people to be able to write adapters without getting their hands dirty with the enumerable convention”. You might find EnumerableRelImplementor.stash solves the problem of passing project lists without generating complex code.

Julian

On Dec 12, 2014, at 11:39 AM, Vladimir Sitnikov <si...@gmail.com> wrote:

> Hi,
> 
> You are looking into Mongo-to-enumerable converter.
> 
> Calcite has 'built-in' execution engine that is called 'enumerable
> convention'.
> It can execute almost all kinds of relational expressions.
> Typically it operates via 'Enumerable' methods.
> To glue this up, a java code is generated on the fly that arranges
> enumerable calls in proper order.
> 
> 'linq4j' is translated to this java code. So, technically, Calcite walks
> this linq4j when preparing java code for enumerable execution.
> 
> Mongo-to-enumerable is needed when you push just a subset of operations to
> mongo and use the result as input to 'enumerable executor'.
> Typically those kind of converters are not expected to include hardcore
> linq4j stuff.
> 
> Does it make sense?
> 
> Can you share the sample of your code that makes you feel bad?
> 
> Regards,
> Vladimir Sitnikov

Re: purpose of using linq4j as an intermediate representation

[Vladimir Sitnikov <si...@gmail.com>]

Hi,

You are looking into Mongo-to-enumerable converter.

Calcite has 'built-in' execution engine that is called 'enumerable
convention'.
It can execute almost all kinds of relational expressions.
Typically it operates via 'Enumerable' methods.
To glue this up, a java code is generated on the fly that arranges
enumerable calls in proper order.

'linq4j' is translated to this java code. So, technically, Calcite walks
this linq4j when preparing java code for enumerable execution.

Mongo-to-enumerable is needed when you push just a subset of operations to
mongo and use the result as input to 'enumerable executor'.
Typically those kind of converters are not expected to include hardcore
linq4j stuff.

Does it make sense?

Can you share the sample of your code that makes you feel bad?

Regards,
Vladimir Sitnikov