[jira] [Comment Edited] (SPARK-11337) Make example code in user guide testable

["Xusen Yin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-11337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14978799#comment-14978799 ] 

Xusen Yin edited comment on SPARK-11337 at 10/28/15 5:25 PM:
-------------------------------------------------------------

I'll create subtasks later. 

One more thing, we need to reorganize the spark/examples later, because there are many small examples that do not need to create a single example file. Take the MLlib datatype page -- mllib-data-types.md -- as an example, code examples like creating vectors and matrices are trivial works. Maybe we can merge them into one single vector/matrix creation example. Then we use labels to distinguish each other, such as {% include_example xxxx.scala vector_creation %}.

The "label way" is also useful in the dialog-style code example: http://spark.apache.org/docs/latest/mllib-linear-methods.html#streaming-linear-regression

So I will skip those trivial examples first and create big ones in the beginning.


was (Author: yinxusen):
I'll create subtasks later. 

One more thing, we need to reorganize the spark/examples later, because there are many small examples that do not need to create a single example file. Take the MLlib datatype page -- mllib-data-types.md -- as an example, code examples like creating vectors and matrices are trivial works. Maybe we can merge them into one single vector/matrix creation example. Then we use labels to distinguish each other, such as {% include_example xxxx.scala vector_creation %}.

The "label way" is also useful in the dialog-style code example: http://spark.apache.org/docs/latest/mllib-linear-methods.html#streaming-linear-regression

So I will skip those trivial examples first and create big ones in the beginning.

> Make example code in user guide testable
> ----------------------------------------
>
>                 Key: SPARK-11337
>                 URL: https://issues.apache.org/jira/browse/SPARK-11337
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Documentation
>            Reporter: Xiangrui Meng
>            Assignee: Xusen Yin
>            Priority: Critical
>
> The example code in the user guide is embedded in the markdown and hence it is not easy to test. It would be nice to automatically test them. This JIRA is to discuss options to automate example code testing and see what we can do in Spark 1.6.
> One option I propose is to move actual example code to spark/examples and test compilation in Jenkins builds. Then in the markdown, we can reference part of the code to show in the user guide. This requires adding a Jekyll tag that is similar to https://github.com/jekyll/jekyll/blob/master/lib/jekyll/tags/include.rb, e.g., called include_example.
> {code}
> {% include_example scala ml.KMeansExample guide %}
> {code}
> Jekyll will find `examples/src/main/scala/org/apache/spark/examples/ml/KMeansExample.scala` and pick code blocks marked "example" and put them under `{% highlight %}` in the markdown. We can discuss the syntax for marker comments.
> Sub-tasks are created to move example code from user guide to `examples/`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org