You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@calcite.apache.org by Abishek Baskaran <ab...@gmail.com> on 2014/09/17 18:14:05 UTC

Filter Push down - passing RexCall from EnumerableRel implement method.

Hi,
I need to push down the filter expression to my custom data source. I see
that Optiq represents the filter expression as a RexCall and I wish to pass
the RexCall object to my custom table. I am able to pass Strings and
Arrays, but unable to figure out how to pass RexCall.

For example, in this codebase
<https://github.com/julianhyde/optiq-csv/blob/master/src/main/java/net/hydromatic/optiq/impl/csv/CsvTableScan.java>
,
We have the implement method that's called on successful push down rule.

  public Result implement(EnumerableRelImplementor implementor, Prefer pref) {
    PhysType physType =
        PhysTypeImpl.of(
            implementor.getTypeFactory(),
            getRowType(),
            pref.preferArray());

    if (table instanceof JsonTable) {
      return implementor.result(
          physType,
          Blocks.toBlock(
              Expressions.call(table.getExpression(JsonTable.class),
                  "enumerable")));
    }
    return implementor.result(
        physType,
        Blocks.toBlock(
            Expressions.call(table.getExpression(CsvTable.class), "project",
                Expressions.constant(fields))));
  }

Expressions.constant(fields) is good for Strings/Arrays. From Splunk
adapter I also figured out how to pass Lists. My requirement is to pass a
RexCall object, would appreciate any guidance here.

Thanks,
Abishek

Re: Filter Push down - passing RexCall from EnumerableRel implement method.

Posted by Abishek Baskaran <ab...@gmail.com>.

Thanks Julian, will try this out and let you know!

On Sun, Sep 21, 2014 at 12:30 AM, Julian Hyde <ju...@gmail.com> wrote:

> Good question. A similar question came up during the hackathon.
>
> The current code generation converts everything to strings. You can’t pass
> an object through — you need to generate java code that will create an
> identical object. That often means serializing the object to something like
> a JSON string and generating a call to de-serialize it.
>
> It’s a bit painful, but it does mean that you can evaluate that code in
> another JVM (even on another machine).
>
> Note that RelJsonWriter can serialize RelNodes to JSON, and RelJsonReader
> can read them back. Take a look at RelWriterTest and you will see that
> there is a way to convert RexNodes to JSON also. (It’s not very easy… you
> could contribute some library methods rexToJson and jsonToRex when you’ve
> figured out how RelJsonWriter and RelJsonReader do it.)
>
> There is a back door which is DataContext. Generated code assumes that
> there is a variable “DataContext root”. That is the entry point for all
> objects (e.g. schemas and open connections) when the plan is running.
> Generated code can call DataContext.get(name) to get an object that has
> been stashed in the context. So, you could stash your RexCall in
> DataContext with a unique name, and generate code that calls something like
>
>   (RexCall) root.get(“name$123”)
>
> In summary, it’s not very easy but it’s possible. Let us know how you get
> on.
>
> Julian
>
>
> On Sep 17, 2014, at 9:14 AM, Abishek Baskaran <ab...@gmail.com>
> wrote:
>
> > Hi,
> > I need to push down the filter expression to my custom data source. I see
> > that Optiq represents the filter expression as a RexCall and I wish to
> pass
> > the RexCall object to my custom table. I am able to pass Strings and
> > Arrays, but unable to figure out how to pass RexCall.
> >
> > For example, in this codebase
> > <
> https://github.com/julianhyde/optiq-csv/blob/master/src/main/java/net/hydromatic/optiq/impl/csv/CsvTableScan.java
> >
> > ,
> > We have the implement method that's called on successful push down rule.
> >
> >  public Result implement(EnumerableRelImplementor implementor, Prefer
> pref) {
> >    PhysType physType =
> >        PhysTypeImpl.of(
> >            implementor.getTypeFactory(),
> >            getRowType(),
> >            pref.preferArray());
> >
> >    if (table instanceof JsonTable) {
> >      return implementor.result(
> >          physType,
> >          Blocks.toBlock(
> >              Expressions.call(table.getExpression(JsonTable.class),
> >                  "enumerable")));
> >    }
> >    return implementor.result(
> >        physType,
> >        Blocks.toBlock(
> >            Expressions.call(table.getExpression(CsvTable.class),
> "project",
> >                Expressions.constant(fields))));
> >  }
> >
> > Expressions.constant(fields) is good for Strings/Arrays. From Splunk
> > adapter I also figured out how to pass Lists. My requirement is to pass a
> > RexCall object, would appreciate any guidance here.
> >
> > Thanks,
> > Abishek
>
>

Re: Filter Push down - passing RexCall from EnumerableRel implement method.

Posted by Julian Hyde <ju...@gmail.com>.

Good question. A similar question came up during the hackathon.

The current code generation converts everything to strings. You can’t pass an object through — you need to generate java code that will create an identical object. That often means serializing the object to something like a JSON string and generating a call to de-serialize it.

It’s a bit painful, but it does mean that you can evaluate that code in another JVM (even on another machine).

Note that RelJsonWriter can serialize RelNodes to JSON, and RelJsonReader can read them back. Take a look at RelWriterTest and you will see that there is a way to convert RexNodes to JSON also. (It’s not very easy… you could contribute some library methods rexToJson and jsonToRex when you’ve figured out how RelJsonWriter and RelJsonReader do it.)

There is a back door which is DataContext. Generated code assumes that there is a variable “DataContext root”. That is the entry point for all objects (e.g. schemas and open connections) when the plan is running. Generated code can call DataContext.get(name) to get an object that has been stashed in the context. So, you could stash your RexCall in DataContext with a unique name, and generate code that calls something like

  (RexCall) root.get(“name$123”)

In summary, it’s not very easy but it’s possible. Let us know how you get on.

Julian

On Sep 17, 2014, at 9:14 AM, Abishek Baskaran <ab...@gmail.com> wrote:

> Hi,
> I need to push down the filter expression to my custom data source. I see
> that Optiq represents the filter expression as a RexCall and I wish to pass
> the RexCall object to my custom table. I am able to pass Strings and
> Arrays, but unable to figure out how to pass RexCall.
> 
> For example, in this codebase
> <https://github.com/julianhyde/optiq-csv/blob/master/src/main/java/net/hydromatic/optiq/impl/csv/CsvTableScan.java>
> ,
> We have the implement method that's called on successful push down rule.
> 
>  public Result implement(EnumerableRelImplementor implementor, Prefer pref) {
>    PhysType physType =
>        PhysTypeImpl.of(
>            implementor.getTypeFactory(),
>            getRowType(),
>            pref.preferArray());
> 
>    if (table instanceof JsonTable) {
>      return implementor.result(
>          physType,
>          Blocks.toBlock(
>              Expressions.call(table.getExpression(JsonTable.class),
>                  "enumerable")));
>    }
>    return implementor.result(
>        physType,
>        Blocks.toBlock(
>            Expressions.call(table.getExpression(CsvTable.class), "project",
>                Expressions.constant(fields))));
>  }
> 
> Expressions.constant(fields) is good for Strings/Arrays. From Splunk
> adapter I also figured out how to pass Lists. My requirement is to pass a
> RexCall object, would appreciate any guidance here.
> 
> Thanks,
> Abishek