You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by "Hartman, Trevor" <th...@ebay.com> on 2015/03/13 21:07:51 UTC
Calcite documentation
I'd like to gauge interest from community members in writing documentation for Calcite. If people are willing to write docs for topics they understand well, I would be willing to help drive that process (collect docs, make sure they're complete, present them cohesively, publish them).
I realize the code is well-documented at the package/class/method level, and while that is great for reference, it's not ideal for understanding big-picture concepts and how subsystems work together.
I've been working with Calcite for nearly 6 months and feel like I've barely scratched the surface in terms of understanding. I think some well-written docs would encourage new users to learn Calcite as well as help existing users take advantage of all Calcite has to offer, which in turn encourages people to blog about it, speak about it, etc.
Suggestions for topics I'm interested in include:
- Overview of Calcite
- Conventions (enumerable, bindable, custom), traits
- Evaluating expression trees
- Pushing down operations, manipulating the relational expression tree
Of course there are may other areas to document that I'm not even aware of, and I'd be looking to you to help define that.
- Are there any Apache standards for docs we should be aware of?
- Where do docs belong? I'm a big fan of in-repo docs written in markdown, viewable on GitHub, but http://calcite.incubator.apache.org might be the more appropriate location.
I looked through jira to see if this work is already being tracked. Nothing comprehensive, but here are some related issues:
https://issues.apache.org/jira/browse/CALCITE-37 Document JSON model file format
https://issues.apache.org/jira/browse/CALCITE-359 Publish javadoc
https://issues.apache.org/jira/browse/CALCITE-355 Create a web site
Trevor
Re: Calcite documentation
Posted by Rajat Venkatesh <rv...@qubole.com>.
Thanks. I'll take a look.
On Wed, Mar 18, 2015 at 12:50 AM, Julian Hyde <ju...@gmail.com> wrote:
>
> > On Mar 17, 2015, at 12:03 AM, Rajat Venkatesh <rv...@qubole.com>
> wrote:
> >
> > Since I am working on this right now:
> >
> > Example for extending the language. I couldnt find examples of client
> > projects setting up the templates and maven targets.
> >
> > I can volunteer to write it as a thank you for guiding me :)
>
> Apache Drill is probably the best example. They extend the SQL grammar and
> use Calcite for optimization. Apache Phoenix is going to do something
> similar, and will probably be simpler, but they have not yet extended the
> SQL grammar.
>
> Julian
--
Rajat Venkatesh | Engg Lead
Qubole Inc | www.qubole.com
Re: Calcite documentation
Posted by Julian Hyde <ju...@gmail.com>.
> On Mar 17, 2015, at 12:03 AM, Rajat Venkatesh <rv...@qubole.com> wrote:
>
> Since I am working on this right now:
>
> Example for extending the language. I couldnt find examples of client
> projects setting up the templates and maven targets.
>
> I can volunteer to write it as a thank you for guiding me :)
Apache Drill is probably the best example. They extend the SQL grammar and use Calcite for optimization. Apache Phoenix is going to do something similar, and will probably be simpler, but they have not yet extended the SQL grammar.
Julian
Re: Calcite documentation
Posted by Rajat Venkatesh <rv...@qubole.com>.
Since I am working on this right now:
Example for extending the language. I couldnt find examples of client
projects setting up the templates and maven targets.
I can volunteer to write it as a thank you for guiding me :)
On Tue, Mar 17, 2015 at 2:10 AM, Hartman, Trevor <th...@ebay.com> wrote:
> > My preferred process would be to treat docs as code, that is, put
> documentation in markdown files in the doc directory, and submit pull
> requests for that doc.
>
> +1 on in-repo PR-driven docs.
>
> Additions to the topic list:
>
> - Writing an adapter
> - Testing an adapter
>
> After removing my items that are covered by Julian's, the full list
> becomes:
>
> (a) Overview of Calcite
> (b) Pushing down operations, manipulating the relational expression tree
> (c) The standard relational expressions
> (d) Overview of planning concepts (RelNode, rule, trait, convention,
> metadata)
> (e) The lifecycle of a query, from parsing to execution
> (f) Enumerable convention (expands on e)
> (g) Bindable convention and the interpreter (expands on e)
> (h) Writing an adapter
> (i) Testing an adapter
>
> Other suggestions for topics?
>
> Trevor
>
--
Rajat Venkatesh | Engg Lead
Qubole Inc | www.qubole.com
Re: Calcite documentation
Posted by "Hartman, Trevor" <th...@ebay.com>.
> My preferred process would be to treat docs as code, that is, put documentation in markdown files in the doc directory, and submit pull requests for that doc.
+1 on in-repo PR-driven docs.
Additions to the topic list:
- Writing an adapter
- Testing an adapter
After removing my items that are covered by Julian's, the full list becomes:
(a) Overview of Calcite
(b) Pushing down operations, manipulating the relational expression tree
(c) The standard relational expressions
(d) Overview of planning concepts (RelNode, rule, trait, convention, metadata)
(e) The lifecycle of a query, from parsing to execution
(f) Enumerable convention (expands on e)
(g) Bindable convention and the interpreter (expands on e)
(h) Writing an adapter
(i) Testing an adapter
Other suggestions for topics?
Trevor
Re: Calcite documentation
Posted by Julian Hyde <ju...@hydromatic.net>.
On Mar 16, 2015, at 10:31 AM, Hartman, Trevor <th...@ebay.com> wrote:
> I created a parent jira item [1] to track all doc-related tasks along with an initial sub-task to define a list of topics [2] with my initial suggestions. I'd like to get some more feedback from committers on fleshing out the topic list.
>
> @Milinda, that looks like a great start. I'm also a fan of Jekyll.
+1 on the site and on Jekyll.
>
> @Julian, please let us know if there are specific processes you prefer (e.g. whether to track docs tasks in jira, where the docs and src website should live—should there be a "calcite" org on github for these sorts of repos?).
There should not be a Calcite org — we are under Apache. Apache has integrated with github to a certain extent — for example, you can submit pull requests to https://github.com/apache/incubator-calcite/pulls, but the process we committers must use to close them is a little more complicated than usual.
My preferred process would be to treat docs as code, that is, put documentation in markdown files in the doc directory, and submit pull requests for that doc. We value contributors who write documentation as highly as those who write code, so regular doc contributors are likely to be invited to become committers.
Moving to jekyll makes sense, because it will allow us to easily deploy our docs as part of the web site.
The mechanism of deploying the web site requires checking files into subversion[1] but only one or two committers need to worry about that. For everyone else, the source of the documentation is git.
Let’s brainstorm on this list to find a list of initial topics and the authors to write them, and we can paste into https://issues.apache.org/jira/browse/CALCITE-622 when we have consensus. To start, here is Trevor’s list:
• (a) Overview of Calcite
• (b) Conventions (enumerable, bindable, custom), traits
• (c) Evaluating expression trees (via compilation, via interpretation)
• (d) Pushing down operations, manipulating the relational expression tree
I would add:
(e) The standard relational expressions
(f) Overview of planning concepts (RelNode, rule, trait, convention, metadata) (covers b)
(g) The lifecycle of a query, from parsing to execution (covers c)
(h) Enumerable convention (expands on g)
(i) Bindable convention and the interpreter (expands on g)
I am not volunteering to write any of these (we tried that strategy, and look where it got us!). If it would help, I’m happy to be “interviewed” over Skype or google hangouts.
Julian
[1] svnpubsub, https://www.apache.org/dev/project-site.html#svnpubsub
Re: Calcite documentation
Posted by "Hartman, Trevor" <th...@ebay.com>.
I created a parent jira item [1] to track all doc-related tasks along with an initial sub-task to define a list of topics [2] with my initial suggestions. I'd like to get some more feedback from committers on fleshing out the topic list.
@Milinda, that looks like a great start. I'm also a fan of Jekyll.
@Julian, please let us know if there are specific processes you prefer (e.g. whether to track docs tasks in jira, where the docs and src website should live—should there be a "calcite" org on github for these sorts of repos?).
[1] https://issues.apache.org/jira/browse/CALCITE-621
[2] https://issues.apache.org/jira/browse/CALCITE-622
Trevor
Re: Calcite documentation
Posted by Milinda Pathirage <mp...@umail.iu.edu>.
Hi All,
I did a quick prototype [1] for Calcite web + documentation. I used
Jekyll[2] for generating [1]. If you prefer something like [1], I can work
on that. Also, please feel free to comment on [1].
Source can be found at [3].
Thanks
Milinda
[1] http://milinda.pathirage.org/calcite-web-prototype/
[2] http://jekyllrb.com/
[3] https://github.com/milinda/calcite-docs-prototype
On Fri, Mar 13, 2015 at 6:53 PM, Julian Hyde <ju...@gmail.com> wrote:
> +1
>
> This is much needed. In my view the best place to start is for people
> to contribute a short article describing a particular task, e.g. "How
> to write a user-defined function".
>
> Format doesn't matter. It could be a Word document or an email message
> in all caps. The editor (say Trevor or whoever commits it) can easily
> convert to the standard format.
>
> The standard format right now is markdown. See the existing docs in
> https://github.com/apache/incubator-calcite/tree/master/doc. The task
> of getting these onto the website is long overdue (mea culpa), but
> that's an issue of presentation, and I think we should focus on
> content.
>
> Julian
>
>
> On Fri, Mar 13, 2015 at 1:28 PM, Colagiovanni, Lawrence
> <lc...@ebay.com> wrote:
> > It'd be awesome if you can make this happen. It sounds like the product
> could really benefit from it, and I know it's an area you're passionate
> about.
> >
> >
> >
> >> On Mar 13, 2015, at 13:09, Hartman, Trevor <th...@ebay.com> wrote:
> >>
> >> I'd like to gauge interest from community members in writing
> documentation for Calcite. If people are willing to write docs for topics
> they understand well, I would be willing to help drive that process
> (collect docs, make sure they're complete, present them cohesively, publish
> them).
> >>
> >> I realize the code is well-documented at the package/class/method
> level, and while that is great for reference, it's not ideal for
> understanding big-picture concepts and how subsystems work together.
> >>
> >> I've been working with Calcite for nearly 6 months and feel like I've
> barely scratched the surface in terms of understanding. I think some
> well-written docs would encourage new users to learn Calcite as well as
> help existing users take advantage of all Calcite has to offer, which in
> turn encourages people to blog about it, speak about it, etc.
> >>
> >> Suggestions for topics I'm interested in include:
> >>
> >> - Overview of Calcite
> >> - Conventions (enumerable, bindable, custom), traits
> >> - Evaluating expression trees
> >> - Pushing down operations, manipulating the relational expression tree
> >>
> >> Of course there are may other areas to document that I'm not even aware
> of, and I'd be looking to you to help define that.
> >>
> >> - Are there any Apache standards for docs we should be aware of?
> >> - Where do docs belong? I'm a big fan of in-repo docs written in
> markdown, viewable on GitHub, but http://calcite.incubator.apache.org
> might be the more appropriate location.
> >>
> >> I looked through jira to see if this work is already being tracked.
> Nothing comprehensive, but here are some related issues:
> >>
> >> https://issues.apache.org/jira/browse/CALCITE-37 Document JSON model
> file format
> >> https://issues.apache.org/jira/browse/CALCITE-359 Publish javadoc
> >> https://issues.apache.org/jira/browse/CALCITE-355 Create a web site
> >>
> >> Trevor
>
--
Milinda Pathirage
PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: Calcite documentation
Posted by Julian Hyde <ju...@gmail.com>.
+1
This is much needed. In my view the best place to start is for people
to contribute a short article describing a particular task, e.g. "How
to write a user-defined function".
Format doesn't matter. It could be a Word document or an email message
in all caps. The editor (say Trevor or whoever commits it) can easily
convert to the standard format.
The standard format right now is markdown. See the existing docs in
https://github.com/apache/incubator-calcite/tree/master/doc. The task
of getting these onto the website is long overdue (mea culpa), but
that's an issue of presentation, and I think we should focus on
content.
Julian
On Fri, Mar 13, 2015 at 1:28 PM, Colagiovanni, Lawrence
<lc...@ebay.com> wrote:
> It'd be awesome if you can make this happen. It sounds like the product could really benefit from it, and I know it's an area you're passionate about.
>
>
>
>> On Mar 13, 2015, at 13:09, Hartman, Trevor <th...@ebay.com> wrote:
>>
>> I'd like to gauge interest from community members in writing documentation for Calcite. If people are willing to write docs for topics they understand well, I would be willing to help drive that process (collect docs, make sure they're complete, present them cohesively, publish them).
>>
>> I realize the code is well-documented at the package/class/method level, and while that is great for reference, it's not ideal for understanding big-picture concepts and how subsystems work together.
>>
>> I've been working with Calcite for nearly 6 months and feel like I've barely scratched the surface in terms of understanding. I think some well-written docs would encourage new users to learn Calcite as well as help existing users take advantage of all Calcite has to offer, which in turn encourages people to blog about it, speak about it, etc.
>>
>> Suggestions for topics I'm interested in include:
>>
>> - Overview of Calcite
>> - Conventions (enumerable, bindable, custom), traits
>> - Evaluating expression trees
>> - Pushing down operations, manipulating the relational expression tree
>>
>> Of course there are may other areas to document that I'm not even aware of, and I'd be looking to you to help define that.
>>
>> - Are there any Apache standards for docs we should be aware of?
>> - Where do docs belong? I'm a big fan of in-repo docs written in markdown, viewable on GitHub, but http://calcite.incubator.apache.org might be the more appropriate location.
>>
>> I looked through jira to see if this work is already being tracked. Nothing comprehensive, but here are some related issues:
>>
>> https://issues.apache.org/jira/browse/CALCITE-37 Document JSON model file format
>> https://issues.apache.org/jira/browse/CALCITE-359 Publish javadoc
>> https://issues.apache.org/jira/browse/CALCITE-355 Create a web site
>>
>> Trevor
Re: Calcite documentation
Posted by "Colagiovanni, Lawrence" <lc...@ebay.com>.
It'd be awesome if you can make this happen. It sounds like the product could really benefit from it, and I know it's an area you're passionate about.
> On Mar 13, 2015, at 13:09, Hartman, Trevor <th...@ebay.com> wrote:
>
> I'd like to gauge interest from community members in writing documentation for Calcite. If people are willing to write docs for topics they understand well, I would be willing to help drive that process (collect docs, make sure they're complete, present them cohesively, publish them).
>
> I realize the code is well-documented at the package/class/method level, and while that is great for reference, it's not ideal for understanding big-picture concepts and how subsystems work together.
>
> I've been working with Calcite for nearly 6 months and feel like I've barely scratched the surface in terms of understanding. I think some well-written docs would encourage new users to learn Calcite as well as help existing users take advantage of all Calcite has to offer, which in turn encourages people to blog about it, speak about it, etc.
>
> Suggestions for topics I'm interested in include:
>
> - Overview of Calcite
> - Conventions (enumerable, bindable, custom), traits
> - Evaluating expression trees
> - Pushing down operations, manipulating the relational expression tree
>
> Of course there are may other areas to document that I'm not even aware of, and I'd be looking to you to help define that.
>
> - Are there any Apache standards for docs we should be aware of?
> - Where do docs belong? I'm a big fan of in-repo docs written in markdown, viewable on GitHub, but http://calcite.incubator.apache.org might be the more appropriate location.
>
> I looked through jira to see if this work is already being tracked. Nothing comprehensive, but here are some related issues:
>
> https://issues.apache.org/jira/browse/CALCITE-37 Document JSON model file format
> https://issues.apache.org/jira/browse/CALCITE-359 Publish javadoc
> https://issues.apache.org/jira/browse/CALCITE-355 Create a web site
>
> Trevor