You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yinxusen <gi...@git.apache.org> on 2015/10/14 06:42:11 UTC
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
GitHub user yinxusen opened a pull request:
https://github.com/apache/spark/pull/9109
[SPARK-10382] [POC] Make example code in user guide testable
A POC code for making example code in user guide testable.
@mengxr We still need to talk about the labels in code.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yinxusen/spark SPARK-10382
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9109.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9109
----
commit f4380f88e0e5d59a1085e69b52e2adc62500d484
Author: Xusen Yin <yi...@gmail.com>
Date: 2015-10-11T06:46:05Z
add code render
commit 0f1d2f94dc8f170432d141e4326280ed86af7721
Author: Xusen Yin <yi...@gmail.com>
Date: 2015-10-13T06:26:34Z
reduce code
commit b0c156866b3136bdcf8c8f43e9facbd2bc36f787
Author: Xusen Yin <yi...@gmail.com>
Date: 2015-10-14T04:37:45Z
add new include example
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42388292
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,70 @@
+require 'octopress-code-highlighter'
+require 'liquid'
+
+module Octopress
+ module IncludeExample
+ class Tag < Liquid::Tag
+
+ # Pattern for {% include_example /path/to/a/file.lan %}
+ FileOnly = /^(\S+)$/
+ # Pattern for {% include_example /path/to/a/file.lan title %}
+ FileTitle = /(\S+)\s+(\S.*?)$/i
+
+ def initialize(tag_name, markup, tokens)
+ @markup = markup
+ super
+ end
+
+ def render(context)
+ site = context.registers[:site]
+ config_dir = (site.config['code_dir'] || '../examples/src/main').sub(/^\//,'')
+ @code_dir = File.join(site.source, config_dir)
+
+ begin
+ options = get_options
+ code = File.open(@file).read.encode("UTF-8")
+ code = select_lines(code, options)
+
+ CodeHighlighter.highlight(code, options)
+ rescue => e
+ puts "not work"
+ end
+ end
+
+ def get_options
+ defaults = {}
+ clean_markup = CodeHighlighter.clean_markup(@markup).strip
+
+ if clean_markup =~ FileOnly
+ @file = File.join(@code_dir, $1)
+ elsif clean_markup =~ FileTitle
+ @file = File.join(@code_dir, $1)
+ defaults[:title] = $2
+ end
+
+ options = CodeHighlighter.parse_markup(@markup, defaults)
+ options[:lang] ||= File.extname(@file).delete('.')
+ options[:link_text] ||= "Example code"
+ options
+ end
+
+ # Select lines according to labels in code. Currently we use "begin code" and "end code" as
+ # labels.
+ def select_lines(code, options)
+ lines = code.each_line.to_a
+ start = lines.each_with_index.select { |l, i| l.include? "begin code" }.last.last
--- End diff --
We don't check the comment symbol like "//" or "#". Just do a string match on the content. So both of the following work. I use `$` just to make sure the text doesn't appear in normal comments.
~~~scala
// $example on$
~~~
~~~python
# $example off$
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42264565
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,70 @@
+require 'octopress-code-highlighter'
+require 'liquid'
+
+module Octopress
+ module IncludeExample
+ class Tag < Liquid::Tag
+
+ # Pattern for {% include_example /path/to/a/file.lan %}
+ FileOnly = /^(\S+)$/
+ # Pattern for {% include_example /path/to/a/file.lan title %}
+ FileTitle = /(\S+)\s+(\S.*?)$/i
+
+ def initialize(tag_name, markup, tokens)
+ @markup = markup
+ super
+ end
+
+ def render(context)
+ site = context.registers[:site]
+ config_dir = (site.config['code_dir'] || '../examples/src/main').sub(/^\//,'')
+ @code_dir = File.join(site.source, config_dir)
+
+ begin
+ options = get_options
+ code = File.open(@file).read.encode("UTF-8")
+ code = select_lines(code, options)
+
+ CodeHighlighter.highlight(code, options)
--- End diff --
Can we generate `{% highlight scala %}` directly? We don't need anything fancy here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149777234
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42264567
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,70 @@
+require 'octopress-code-highlighter'
+require 'liquid'
+
+module Octopress
+ module IncludeExample
+ class Tag < Liquid::Tag
+
+ # Pattern for {% include_example /path/to/a/file.lan %}
+ FileOnly = /^(\S+)$/
+ # Pattern for {% include_example /path/to/a/file.lan title %}
+ FileTitle = /(\S+)\s+(\S.*?)$/i
+
+ def initialize(tag_name, markup, tokens)
+ @markup = markup
+ super
+ end
+
+ def render(context)
+ site = context.registers[:site]
+ config_dir = (site.config['code_dir'] || '../examples/src/main').sub(/^\//,'')
+ @code_dir = File.join(site.source, config_dir)
+
+ begin
+ options = get_options
+ code = File.open(@file).read.encode("UTF-8")
+ code = select_lines(code, options)
+
+ CodeHighlighter.highlight(code, options)
+ rescue => e
+ puts "not work"
+ end
+ end
+
+ def get_options
+ defaults = {}
+ clean_markup = CodeHighlighter.clean_markup(@markup).strip
+
+ if clean_markup =~ FileOnly
+ @file = File.join(@code_dir, $1)
+ elsif clean_markup =~ FileTitle
+ @file = File.join(@code_dir, $1)
+ defaults[:title] = $2
+ end
+
+ options = CodeHighlighter.parse_markup(@markup, defaults)
+ options[:lang] ||= File.extname(@file).delete('.')
+ options[:link_text] ||= "Example code"
+ options
+ end
+
+ # Select lines according to labels in code. Currently we use "begin code" and "end code" as
+ # labels.
+ def select_lines(code, options)
+ lines = code.each_line.to_a
+ start = lines.each_with_index.select { |l, i| l.include? "begin code" }.last.last
--- End diff --
"begin code" and "end code" are quite common. Shall we switch to `$example on$` and `$example off$`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149773183
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42840074
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,79 @@
+require 'liquid'
+require 'pygments'
+
+module Jekyll
+ class IncludeExampleTag < Liquid::Tag
+
+ def initialize(tag_name, markup, tokens)
+ @markup = markup
+ super
+ end
+
+ def render(context)
+ site = context.registers[:site]
+ config_dir = (site.config['code_dir'] || '../examples/src/main').sub(/^\//,'')
+ @code_dir = File.join(site.source, config_dir)
+
+ clean_markup = @markup.strip
+ @file = File.join(@code_dir, clean_markup)
+ @lang = clean_markup.split('.').last
+
+ code = File.open(@file).read.encode("UTF-8")
+ code = select_lines(code)
+
+ Pygments.highlight(code, :lexer => @lang)
+ end
+
+ # Trim the code block so as to have the same indention, regardless of their positions in the
+ # code file.
+ def trim_codeblock(lines)
+ # Select the minimum indention of the current code block.
+ min_start_spaces = lines
+ .select{ |l| l.strip.size !=0 }
--- End diff --
Sure, will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42838120
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,79 @@
+require 'liquid'
--- End diff --
Could you add Apache header? See `copy_api_dirs.rb`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150504956
**[Test build #44212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44212/consoleFull)** for PR 9109 at commit [`8557977`](https://github.com/apache/spark/commit/85579771df81440d4a67b787e5bff298dd6346c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42838196
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,79 @@
+require 'liquid'
+require 'pygments'
+
+module Jekyll
+ class IncludeExampleTag < Liquid::Tag
+
+ def initialize(tag_name, markup, tokens)
+ @markup = markup
+ super
+ end
+
+ def render(context)
+ site = context.registers[:site]
+ config_dir = (site.config['code_dir'] || '../examples/src/main').sub(/^\//,'')
+ @code_dir = File.join(site.source, config_dir)
+
+ clean_markup = @markup.strip
+ @file = File.join(@code_dir, clean_markup)
+ @lang = clean_markup.split('.').last
+
+ code = File.open(@file).read.encode("UTF-8")
+ code = select_lines(code)
+
+ Pygments.highlight(code, :lexer => @lang)
+ end
+
+ # Trim the code block so as to have the same indention, regardless of their positions in the
+ # code file.
+ def trim_codeblock(lines)
+ # Select the minimum indention of the current code block.
+ min_start_spaces = lines
+ .select{ |l| l.strip.size !=0 }
--- End diff --
nit: if we still want to follow Spark code style, it should be `select {` instead of `select{`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150622432
@mengxr Sure I can do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149758646
@mengxr Yes, I'll do it ASAP.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149777235
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44036/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150493909
Yes, it looks good. Please remove the example, add Apache header, then remove `[POC]` from the PR title. It should be good to go.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150609532
LGTM. Merged into master. Thanks! This makes the example code much easier to check. Could you make one JIRA and submit a PR to replace some example code using this? Then we can create more JIRAs, and ask community to help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149302530
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43932/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149297856
It requires Pygments. So install Pygments before compile it. Actually it is the same with what defined in _config.yml.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-147934833
[Test build #43703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43703/consoleFull) for PR 9109 at commit [`b0c1568`](https://github.com/apache/spark/commit/b0c156866b3136bdcf8c8f43e9facbd2bc36f787).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150505200
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149338997
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43931/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149302527
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-147937024
[Test build #43703 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43703/console) for PR 9109 at commit [`b0c1568`](https://github.com/apache/spark/commit/b0c156866b3136bdcf8c8f43e9facbd2bc36f787).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149772850
@mengxr You can check the POC now. If it is OK, I will remove the example. Next time we can enjoy easy code sample insertion in docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-147934070
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149338793
**[Test build #43931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43931/consoleFull)** for PR 9109 at commit [`830d1b9`](https://github.com/apache/spark/commit/830d1b92605bcfd6f724e0bbd51a17681b6462de).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149300401
**[Test build #43931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43931/consoleFull)** for PR 9109 at commit [`830d1b9`](https://github.com/apache/spark/commit/830d1b92605bcfd6f724e0bbd51a17681b6462de).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149773234
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149342077
@yinxusen It is possible that we only replace the markdown text and let `highlight` handle the syntax highlighting? Another minor comment is removing the `// $example on$` and `// $example off$` in the output file. Could you also include the imports in the example code? Basically, having two example blocks from the source file. Just want to make sure it works.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42264562
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,70 @@
+require 'octopress-code-highlighter'
+require 'liquid'
+
+module Octopress
--- End diff --
If our implementation doesn't require octopress, we should rename it and cite it in the reference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149777177
**[Test build #44036 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44036/consoleFull)** for PR 9109 at commit [`78c07ae`](https://github.com/apache/spark/commit/78c07aebd3c8c09c61a45f7c0353d4e8a4bb682d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149297433
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149298223
**[Test build #43932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43932/consoleFull)** for PR 9109 at commit [`0dbcfe9`](https://github.com/apache/spark/commit/0dbcfe9014a05856b1e97daf8a183d8fbda2cd2a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149774265
**[Test build #44036 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44036/consoleFull)** for PR 9109 at commit [`78c07ae`](https://github.com/apache/spark/commit/78c07aebd3c8c09c61a45f7c0353d4e8a4bb682d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150501534
**[Test build #44212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44212/consoleFull)** for PR 9109 at commit [`8557977`](https://github.com/apache/spark/commit/85579771df81440d4a67b787e5bff298dd6346c8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150505202
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44212/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42306886
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,70 @@
+require 'octopress-code-highlighter'
+require 'liquid'
+
+module Octopress
+ module IncludeExample
+ class Tag < Liquid::Tag
+
+ # Pattern for {% include_example /path/to/a/file.lan %}
+ FileOnly = /^(\S+)$/
+ # Pattern for {% include_example /path/to/a/file.lan title %}
+ FileTitle = /(\S+)\s+(\S.*?)$/i
+
+ def initialize(tag_name, markup, tokens)
+ @markup = markup
+ super
+ end
+
+ def render(context)
+ site = context.registers[:site]
+ config_dir = (site.config['code_dir'] || '../examples/src/main').sub(/^\//,'')
+ @code_dir = File.join(site.source, config_dir)
+
+ begin
+ options = get_options
+ code = File.open(@file).read.encode("UTF-8")
+ code = select_lines(code, options)
+
+ CodeHighlighter.highlight(code, options)
--- End diff --
So there is no need to call outside function, just translate it into markdown highlight. Then we can avoid to use Octopress in the beginning. I get it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149727620
@yinxusen Could you update the PR to support multiple example code blocks in one file? It is quite comment to have imports + example code - boilerplate code in the user guide. If you show that it works, we can remove the example and then merge this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149338996
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149297475
@mengxr I add a POC example of ml#Pipeline. But I am not sure that I select right files. But it demos what we need.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149297488
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42306906
--- Diff: docs/index.md ---
@@ -12,6 +12,8 @@ It also supports a rich set of higher-level tools including [Spark SQL](sql-prog
# Downloading
+{% include_example scala/org/apache/spark/examples/SparkALS.scala SparkALS %}
--- End diff --
Sure I will add it after refactoring the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149302363
**[Test build #43932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43932/consoleFull)** for PR 9109 at commit [`0dbcfe9`](https://github.com/apache/spark/commit/0dbcfe9014a05856b1e97daf8a183d8fbda2cd2a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9109
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149296211
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-149296263
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42264578
--- Diff: docs/index.md ---
@@ -12,6 +12,8 @@ It also supports a rich set of higher-level tools including [Spark SQL](sql-prog
# Downloading
+{% include_example scala/org/apache/spark/examples/SparkALS.scala SparkALS %}
--- End diff --
Could turn it into a real example including Scala/Java/Python? We can use the pipeline example http://spark.apache.org/docs/latest/ml-guide.html#example-pipeline.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-147934064
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150500190
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] Make example code in user guide ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9109#issuecomment-150500211
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42306903
--- Diff: docs/_plugins/include_example.rb ---
@@ -0,0 +1,70 @@
+require 'octopress-code-highlighter'
+require 'liquid'
+
+module Octopress
+ module IncludeExample
+ class Tag < Liquid::Tag
+
+ # Pattern for {% include_example /path/to/a/file.lan %}
+ FileOnly = /^(\S+)$/
+ # Pattern for {% include_example /path/to/a/file.lan title %}
+ FileTitle = /(\S+)\s+(\S.*?)$/i
+
+ def initialize(tag_name, markup, tokens)
+ @markup = markup
+ super
+ end
+
+ def render(context)
+ site = context.registers[:site]
+ config_dir = (site.config['code_dir'] || '../examples/src/main').sub(/^\//,'')
+ @code_dir = File.join(site.source, config_dir)
+
+ begin
+ options = get_options
+ code = File.open(@file).read.encode("UTF-8")
+ code = select_lines(code, options)
+
+ CodeHighlighter.highlight(code, options)
+ rescue => e
+ puts "not work"
+ end
+ end
+
+ def get_options
+ defaults = {}
+ clean_markup = CodeHighlighter.clean_markup(@markup).strip
+
+ if clean_markup =~ FileOnly
+ @file = File.join(@code_dir, $1)
+ elsif clean_markup =~ FileTitle
+ @file = File.join(@code_dir, $1)
+ defaults[:title] = $2
+ end
+
+ options = CodeHighlighter.parse_markup(@markup, defaults)
+ options[:lang] ||= File.extname(@file).delete('.')
+ options[:link_text] ||= "Example code"
+ options
+ end
+
+ # Select lines according to labels in code. Currently we use "begin code" and "end code" as
+ # labels.
+ def select_lines(code, options)
+ lines = code.each_line.to_a
+ start = lines.each_with_index.select { |l, i| l.include? "begin code" }.last.last
--- End diff --
Here is the thing, different language requires different comment signs. So in Java and Scala we have `// example on` and `// example off` (or `/* example on/off */`) and in Python, it is `# example on/off`. Does your `$example on/off$` mean that we use the dollar sign to distinguish our label with common code comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10382] [POC] Make example code in user ...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9109#discussion_r42264560
--- Diff: docs/_config.yml ---
@@ -2,6 +2,7 @@ highlighter: pygments
markdown: kramdown
gems:
- jekyll-redirect-from
+ - octopress-code-highlighter
--- End diff --
Do we still need this gem if we have `include_example.rb`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org