Beam SQL: Integrating runners & IO

[Kenneth Knowles <kl...@google.com>]

Hi all,

Andrew mentioned something super cool about what he and Anton have done
with Beam SQL: it now implements a JDBC driver (via Calcite Avatica).

And since the sqlline client is tiny, it is just baked in. So you can java
-jar the JDBC driver and run a little shell. But the shell only has the
direct runner, and to make a nice experience for SQL users who don't want
to deal with Java, there are some challenges.

I wrote up a really quick doc for brainstorming

https://s.apache.org/beam-sql-packaging

Please lend you comments and expertise!

Kenn

Re: Beam SQL: Integrating runners & IO

[Andrew Pilloud <ap...@google.com>]

Thanks for the great writeup Kenn! I really like the part about pushing
the TableProvider abstraction into core and the IOs. This would make it
really easy to extend the IOs supported by your SQL shell just by adding
the appropriate IOs to the classpath. It would also make testing much
easier as we could create mock IOs that don't depend on SQL for some of the
JDBC integration tests.

Andrew

On Mon, Jun 11, 2018 at 10:44 AM Kenneth Knowles <kl...@google.com> wrote:

> Hi all,
>
> Andrew mentioned something super cool about what he and Anton have done
> with Beam SQL: it now implements a JDBC driver (via Calcite Avatica).
>
> And since the sqlline client is tiny, it is just baked in. So you can java
> -jar the JDBC driver and run a little shell. But the shell only has the
> direct runner, and to make a nice experience for SQL users who don't want
> to deal with Java, there are some challenges.
>
> I wrote up a really quick doc for brainstorming
>
> https://s.apache.org/beam-sql-packaging
>
> Please lend you comments and expertise!
>
> Kenn
>