Shared enumerator

[Liam Fisk <li...@xtra.co.nz>]

Hi,

Is it possible to have a set of tables backed by a shared enumerator, and let Calcite do its thing as usual?

I'm wanting to project a document with a mix of scalars and arrays into say 3 tables, each with a different cardinality, and each allowing filters to be pushed down. Each table would have their "own" Calcite enumerator, due to the mismatched cardinalities, but these enumerators would be backed by a single shared iterator fetching documents. A fetch could only occur once each enumerator had exhausted the rows extracted from the current document, and thus all enumerators had signalled "next()".

This pattern would avoid having 3 tables created from different views of the underlying document, and this would avoid multiple network round trips. Can this be achieved with Calcite, or do the operations on each table need to be completely isolated?

Thanks,

Liam

Re: Shared enumerator

[Julian Hyde <ju...@gmail.com>]

I presume that you are reading from some source and using ENUMERABLE convention, so that each table scan ultimately turns into an Enumerable.enumerator() call?

If so, it doesn’t sound too difficult. You can make sure that the Enumerable that implements your table actually looks in a cache first.

I suggest that you run your query with -Dcalcite.debug=true so that you can see the Java code that is being generated. Find the piece of that code that reads from the table, and figure out how to convert it to read from a cache. (You may need to write an implementation of Enumerator that caches rows in a backing array.) Then you can change generator code, which is probably the

  Expression getExpression(SchemaPlus schema, String tableName, Class clazz)

method of your Table implementation.

Julian

On Mar 2, 2015, at 2:19 AM, Liam Fisk <li...@xtra.co.nz> wrote:

> Hi,
> 
> Is it possible to have a set of tables backed by a shared enumerator, and let Calcite do its thing as usual?
> 
> I'm wanting to project a document with a mix of scalars and arrays into say 3 tables, each with a different cardinality, and each allowing filters to be pushed down. Each table would have their "own" Calcite enumerator, due to the mismatched cardinalities, but these enumerators would be backed by a single shared iterator fetching documents. A fetch could only occur once each enumerator had exhausted the rows extracted from the current document, and thus all enumerators had signalled "next()".
> 
> This pattern would avoid having 3 tables created from different views of the underlying document, and this would avoid multiple network round trips. Can this be achieved with Calcite, or do the operations on each table need to be completely isolated?
> 
> Thanks,
> 
> Liam
>