You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Michael Mior <mm...@apache.org> on 2018/12/19 18:52:53 UTC
Calcite example code
After seeing so many people ask for example code to do certain basic things
in Calcite, I've been trying to find a good literate programming solution
for Java as I like this approach for demoing. I recently came across the
IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks.
This is basically just a proof of concept at this point, but here's a
simple example
https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
I'm curious what others think of this approach. If others think it would be
useful, I'd be happy to take suggestions on what should be included.
Eventually, I'd like to get CI set up for this repository so I can re-run
the notebooks at will. I would then aim to check this on every release so
we can have a repository of code samples which we know run correctly.
--
Michael Mior
mmior@apache.org
Re: Calcite example code
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
Michael,
I would definitely encourage you and anyone else to participate. That is an
awesome idea, especially sometimes there are multiple ways to do the same
thing. For example, instantiating a RelNode converter can be done by the
planner or by constructing everything by hand. Similarly adding schemas can
be done programmatically or by model files. Pointing out the situations of
when to use each solution would be a great addition to this.
Thanks,
Walaa.
On Wed, Dec 19, 2018 at 10:53 AM Michael Mior <mm...@apache.org> wrote:
> After seeing so many people ask for example code to do certain basic things
> in Calcite, I've been trying to find a good literate programming solution
> for Java as I like this approach for demoing. I recently came across the
> IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks.
>
> This is basically just a proof of concept at this point, but here's a
> simple example
>
>
> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
>
> I'm curious what others think of this approach. If others think it would be
> useful, I'd be happy to take suggestions on what should be included.
> Eventually, I'd like to get CI set up for this repository so I can re-run
> the notebooks at will. I would then aim to check this on every release so
> we can have a repository of code samples which we know run correctly.
>
> --
> Michael Mior
> mmior@apache.org
>
Re: Calcite example code
Posted by Kevin Risden <kr...@apache.org>.
Vladimir since on a different thread you asked about playground.
As a side note, it might be interesting to have Calcite playground.
> It could be like https://rextester.com/l/postgresql_online_compiler but
> with Calcite inside.
>
Michael had setup one with jupyter and binder. Its not exactly the same
thing, but definitely allows playing around with Calcite.
https://github.com/michaelmior/calcite-notebooks
Kevin Risden
On Thu, Dec 20, 2018 at 12:48 PM Michael Mior <mm...@apache.org> wrote:
> Here's another one focused on basic query optimization
>
>
> https://github.com/michaelmior/calcite-notebooks/blob/master/query-optimization.ipynb
>
> --
> Michael Mior
> mmior@apache.org
>
>
> Le mer. 19 déc. 2018 à 18:21, Michael Mior <mm...@apache.org> a écrit :
>
> > Yes the notebook contains the output (it's really just a JSON file). It
> > doesn't necessarily have to have output, but it's much more useful if it
> > does since it means whoever is viewing doesn't need to execute it. It
> would
> > certainly be possible to use this for tests although it would require an
> > installation of Python. Given how ubiquitous Python is, I don't think
> this
> > is a huge concern, although we'd need a way of installing a couple Python
> > dependencies.
> >
> > As Kevin mentioned, you can execute these online with Binder. I just had
> > to add a Dockerfile so it could run the IJava kernel since it not the
> > default. Check out the link below and you can see what the experience is
> > like.
> >
> > https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master
> >
> > Just select a notebook once it loads (it may take a couple minutes). The
> > experience is basically the same as what you get when running locally.
> The
> > notebook consists of a series of "cells" which you can run individually
> and
> > edit as you wish. This would also make it easy for people to play around
> > with Calcite a little without having to install anything.
> >
> > --
> > Michael Mior
> > mmior@apache.org
> >
> >
> > Le mer. 19 déc. 2018 à 17:22, Julian Hyde <jh...@apache.org> a écrit :
> >
> >> For old idiots like me, can you explain how the notebook works? The file
> >> you checked into GitHub, does it contain the input and output or just
> the
> >> input? Is there a way to edit or use the notebook interactively?
> >>
> >> It certainly seems a better way to introduce people to examples than
> >> saying “go look at this test”.
> >>
> >> I think quite a few of our tests could be converted into this format.
> >>
> >> Julian
> >>
> >>
> >> > On Dec 19, 2018, at 10:52 AM, Michael Mior <mm...@apache.org> wrote:
> >> >
> >> > After seeing so many people ask for example code to do certain basic
> >> things
> >> > in Calcite, I've been trying to find a good literate programming
> >> solution
> >> > for Java as I like this approach for demoing. I recently came across
> the
> >> > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter
> >> notebooks.
> >> >
> >> > This is basically just a proof of concept at this point, but here's a
> >> > simple example
> >> >
> >> >
> >>
> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
> >> >
> >> > I'm curious what others think of this approach. If others think it
> >> would be
> >> > useful, I'd be happy to take suggestions on what should be included.
> >> > Eventually, I'd like to get CI set up for this repository so I can
> >> re-run
> >> > the notebooks at will. I would then aim to check this on every release
> >> so
> >> > we can have a repository of code samples which we know run correctly.
> >> >
> >> > --
> >> > Michael Mior
> >> > mmior@apache.org
> >>
> >>
>
Re: Calcite example code
Posted by Michael Mior <mm...@apache.org>.
Here's another one focused on basic query optimization
https://github.com/michaelmior/calcite-notebooks/blob/master/query-optimization.ipynb
--
Michael Mior
mmior@apache.org
Le mer. 19 déc. 2018 à 18:21, Michael Mior <mm...@apache.org> a écrit :
> Yes the notebook contains the output (it's really just a JSON file). It
> doesn't necessarily have to have output, but it's much more useful if it
> does since it means whoever is viewing doesn't need to execute it. It would
> certainly be possible to use this for tests although it would require an
> installation of Python. Given how ubiquitous Python is, I don't think this
> is a huge concern, although we'd need a way of installing a couple Python
> dependencies.
>
> As Kevin mentioned, you can execute these online with Binder. I just had
> to add a Dockerfile so it could run the IJava kernel since it not the
> default. Check out the link below and you can see what the experience is
> like.
>
> https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master
>
> Just select a notebook once it loads (it may take a couple minutes). The
> experience is basically the same as what you get when running locally. The
> notebook consists of a series of "cells" which you can run individually and
> edit as you wish. This would also make it easy for people to play around
> with Calcite a little without having to install anything.
>
> --
> Michael Mior
> mmior@apache.org
>
>
> Le mer. 19 déc. 2018 à 17:22, Julian Hyde <jh...@apache.org> a écrit :
>
>> For old idiots like me, can you explain how the notebook works? The file
>> you checked into GitHub, does it contain the input and output or just the
>> input? Is there a way to edit or use the notebook interactively?
>>
>> It certainly seems a better way to introduce people to examples than
>> saying “go look at this test”.
>>
>> I think quite a few of our tests could be converted into this format.
>>
>> Julian
>>
>>
>> > On Dec 19, 2018, at 10:52 AM, Michael Mior <mm...@apache.org> wrote:
>> >
>> > After seeing so many people ask for example code to do certain basic
>> things
>> > in Calcite, I've been trying to find a good literate programming
>> solution
>> > for Java as I like this approach for demoing. I recently came across the
>> > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter
>> notebooks.
>> >
>> > This is basically just a proof of concept at this point, but here's a
>> > simple example
>> >
>> >
>> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
>> >
>> > I'm curious what others think of this approach. If others think it
>> would be
>> > useful, I'd be happy to take suggestions on what should be included.
>> > Eventually, I'd like to get CI set up for this repository so I can
>> re-run
>> > the notebooks at will. I would then aim to check this on every release
>> so
>> > we can have a repository of code samples which we know run correctly.
>> >
>> > --
>> > Michael Mior
>> > mmior@apache.org
>>
>>
Re: Calcite example code
Posted by Michael Mior <mm...@apache.org>.
Yes the notebook contains the output (it's really just a JSON file). It
doesn't necessarily have to have output, but it's much more useful if it
does since it means whoever is viewing doesn't need to execute it. It would
certainly be possible to use this for tests although it would require an
installation of Python. Given how ubiquitous Python is, I don't think this
is a huge concern, although we'd need a way of installing a couple Python
dependencies.
As Kevin mentioned, you can execute these online with Binder. I just had to
add a Dockerfile so it could run the IJava kernel since it not the default.
Check out the link below and you can see what the experience is like.
https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master
Just select a notebook once it loads (it may take a couple minutes). The
experience is basically the same as what you get when running locally. The
notebook consists of a series of "cells" which you can run individually and
edit as you wish. This would also make it easy for people to play around
with Calcite a little without having to install anything.
--
Michael Mior
mmior@apache.org
Le mer. 19 déc. 2018 à 17:22, Julian Hyde <jh...@apache.org> a écrit :
> For old idiots like me, can you explain how the notebook works? The file
> you checked into GitHub, does it contain the input and output or just the
> input? Is there a way to edit or use the notebook interactively?
>
> It certainly seems a better way to introduce people to examples than
> saying “go look at this test”.
>
> I think quite a few of our tests could be converted into this format.
>
> Julian
>
>
> > On Dec 19, 2018, at 10:52 AM, Michael Mior <mm...@apache.org> wrote:
> >
> > After seeing so many people ask for example code to do certain basic
> things
> > in Calcite, I've been trying to find a good literate programming solution
> > for Java as I like this approach for demoing. I recently came across the
> > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter
> notebooks.
> >
> > This is basically just a proof of concept at this point, but here's a
> > simple example
> >
> >
> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
> >
> > I'm curious what others think of this approach. If others think it would
> be
> > useful, I'd be happy to take suggestions on what should be included.
> > Eventually, I'd like to get CI set up for this repository so I can re-run
> > the notebooks at will. I would then aim to check this on every release so
> > we can have a repository of code samples which we know run correctly.
> >
> > --
> > Michael Mior
> > mmior@apache.org
>
>
Re: Calcite example code
Posted by Francis Chuang <fr...@apache.org>.
+1 for jupyter notebooks
@Julian:
I have not used Jupyter notebooks myself, but read about them a few
months ago.
My understanding is that you can open the ipynb files using the
opensource Jupyter web app and change the code interactively using code
editors or widgets such as sliders etc. The changes would then be
reflected back in the interface in real-time.
I suppose it would be similar to documented examples in Go:
https://godoc.org/fmt#Stringer (click example to expand it, then click
play to open the code in the playground), but with much more
interactivity and a better user experience
On 20/12/2018 9:22 am, Julian Hyde wrote:
> For old idiots like me, can you explain how the notebook works? The file you checked into GitHub, does it contain the input and output or just the input? Is there a way to edit or use the notebook interactively?
>
> It certainly seems a better way to introduce people to examples than saying “go look at this test”.
>
> I think quite a few of our tests could be converted into this format.
>
> Julian
>
>
>> On Dec 19, 2018, at 10:52 AM, Michael Mior <mm...@apache.org> wrote:
>>
>> After seeing so many people ask for example code to do certain basic things
>> in Calcite, I've been trying to find a good literate programming solution
>> for Java as I like this approach for demoing. I recently came across the
>> IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks.
>>
>> This is basically just a proof of concept at this point, but here's a
>> simple example
>>
>> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
>>
>> I'm curious what others think of this approach. If others think it would be
>> useful, I'd be happy to take suggestions on what should be included.
>> Eventually, I'd like to get CI set up for this repository so I can re-run
>> the notebooks at will. I would then aim to check this on every release so
>> we can have a repository of code samples which we know run correctly.
>>
>> --
>> Michael Mior
>> mmior@apache.org
Re: Calcite example code
Posted by Kevin Risden <kr...@apache.org>.
I like the idea a lot.
Notebooks are interactive ways to work with code and results. The
notebook typically contains both input and output although you can
save it without the output. The notebook can be used interactively
with Jupyter or one of the options below.
I was going to share this as well since it helps with notebooks on Github:
Static viewing
* https://nbviewer.jupyter.org/
* https://nbviewer.jupyter.org/github/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb#
Dynamic notebook
* https://mybinder.org/
* https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master
The dynamic notebook could use a little work to get the Java kernel to
work correctly but it is definitely possible.
Kevin Risden
On Wed, Dec 19, 2018 at 5:22 PM Julian Hyde <jh...@apache.org> wrote:
>
> For old idiots like me, can you explain how the notebook works? The file you checked into GitHub, does it contain the input and output or just the input? Is there a way to edit or use the notebook interactively?
>
> It certainly seems a better way to introduce people to examples than saying “go look at this test”.
>
> I think quite a few of our tests could be converted into this format.
>
> Julian
>
>
> > On Dec 19, 2018, at 10:52 AM, Michael Mior <mm...@apache.org> wrote:
> >
> > After seeing so many people ask for example code to do certain basic things
> > in Calcite, I've been trying to find a good literate programming solution
> > for Java as I like this approach for demoing. I recently came across the
> > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks.
> >
> > This is basically just a proof of concept at this point, but here's a
> > simple example
> >
> > https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
> >
> > I'm curious what others think of this approach. If others think it would be
> > useful, I'd be happy to take suggestions on what should be included.
> > Eventually, I'd like to get CI set up for this repository so I can re-run
> > the notebooks at will. I would then aim to check this on every release so
> > we can have a repository of code samples which we know run correctly.
> >
> > --
> > Michael Mior
> > mmior@apache.org
>
Re: Calcite example code
Posted by Julian Hyde <jh...@apache.org>.
For old idiots like me, can you explain how the notebook works? The file you checked into GitHub, does it contain the input and output or just the input? Is there a way to edit or use the notebook interactively?
It certainly seems a better way to introduce people to examples than saying “go look at this test”.
I think quite a few of our tests could be converted into this format.
Julian
> On Dec 19, 2018, at 10:52 AM, Michael Mior <mm...@apache.org> wrote:
>
> After seeing so many people ask for example code to do certain basic things
> in Calcite, I've been trying to find a good literate programming solution
> for Java as I like this approach for demoing. I recently came across the
> IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks.
>
> This is basically just a proof of concept at this point, but here's a
> simple example
>
> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb
>
> I'm curious what others think of this approach. If others think it would be
> useful, I'd be happy to take suggestions on what should be included.
> Eventually, I'd like to get CI set up for this repository so I can re-run
> the notebooks at will. I would then aim to check this on every release so
> we can have a repository of code samples which we know run correctly.
>
> --
> Michael Mior
> mmior@apache.org