You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by pr...@apache.org on 2019/09/23 04:45:24 UTC
[zeppelin] branch master updated: [ZEPPELIN-4311] Supporting new parser for markdown as pegdown parser …
This is an automated email from the ASF dual-hosted git repository.
prabhjyotsingh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 0d9239e [ZEPPELIN-4311] Supporting new parser for markdown as pegdown parser …
0d9239e is described below
commit 0d9239ef333e04e0788dbccfd1d00314b97a4370
Author: Bhavik Patel <bh...@gmail.com>
AuthorDate: Tue Sep 17 12:30:47 2019 +0530
[ZEPPELIN-4311] Supporting new parser for markdown as pegdown parser …
### What is this PR for?
Supporting new parser flexmark for markdown, as pegdown parser is deprecated and facing issue with pegdown parser.
### What type of PR is it?
[Bug Fix | Improvement]
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4311
### How should this be tested?
CI Pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Bhavik Patel <bh...@gmail.com>
Closes #3443 from bhavikpatel9977/ZEPPELIN-4311 and squashes the following commits:
59edd8d73 [Bhavik Patel] Incroporated the review changes
0f5980381 [Bhavik Patel] added license and documentation
e28f68226 [Bhavik Patel] Added license
e4b54ce4a [Bhavik Patel] [ZEPPELIN-4311] Supporting new parser for markdown as pegdown parser is deprecated and facing issue with parser
Change-Id: I4aa3bc93267b3a5d5003af176f84aa37e4c742ff
---
LICENSE | 2 +
...markdown-example-flexmark-parser-extensions.png | Bin 0 -> 41115 bytes
.../docs-img/markdown-example-flexmark-parser.png | Bin 0 -> 31019 bytes
docs/interpreter/markdown.md | 20 +-
markdown/README.md | 34 +++
markdown/pom.xml | 7 +
.../apache/zeppelin/markdown/FlexmarkParser.java | 73 +++++++
.../org/apache/zeppelin/markdown/Markdown.java | 12 +-
.../apache/zeppelin/markdown/UMLBlockQuote.java | 118 +++++++++++
.../zeppelin/markdown/UMLBlockQuoteParser.java | 187 +++++++++++++++++
.../org/apache/zeppelin/markdown/UMLExtension.java | 57 +++++
.../apache/zeppelin/markdown/UMLNodeRenderer.java | 231 +++++++++++++++++++++
.../src/main/resources/interpreter-setting.json | 4 +-
.../zeppelin/markdown/FlexmarkParserTest.java | 225 ++++++++++++++++++++
14 files changed, 959 insertions(+), 11 deletions(-)
diff --git a/LICENSE b/LICENSE
index 40c1143..121d2e1 100644
--- a/LICENSE
+++ b/LICENSE
@@ -287,6 +287,8 @@ The following components are provided under the BSD 2-Clause license. See file
(BSD 2 Clause) portions of SQLLine (http://sqlline.sourceforge.net/) - http://sqlline.sourceforge.net/#license
jdbc/src/main/java/org/apache/zeppelin/jdbc/SqlCompleter.java
+ (BSD 2-clause) flexmark-java (https://github.com/vsch/flexmark-java)
+
========================================================================
Jython Software License
========================================================================
diff --git a/docs/assets/themes/zeppelin/img/docs-img/markdown-example-flexmark-parser-extensions.png b/docs/assets/themes/zeppelin/img/docs-img/markdown-example-flexmark-parser-extensions.png
new file mode 100644
index 0000000..8af5c9e
Binary files /dev/null and b/docs/assets/themes/zeppelin/img/docs-img/markdown-example-flexmark-parser-extensions.png differ
diff --git a/docs/assets/themes/zeppelin/img/docs-img/markdown-example-flexmark-parser.png b/docs/assets/themes/zeppelin/img/docs-img/markdown-example-flexmark-parser.png
new file mode 100644
index 0000000..87f9faa
Binary files /dev/null and b/docs/assets/themes/zeppelin/img/docs-img/markdown-example-flexmark-parser.png differ
diff --git a/docs/interpreter/markdown.md b/docs/interpreter/markdown.md
index 609f204..e06c563 100644
--- a/docs/interpreter/markdown.md
+++ b/docs/interpreter/markdown.md
@@ -25,7 +25,7 @@ limitations under the License.
## Overview
[Markdown](http://daringfireball.net/projects/markdown/) is a plain text formatting syntax designed so that it can be converted to HTML.
-Apache Zeppelin uses [pegdown](https://github.com/sirthias/pegdown) and [markdown4j](https://github.com/jdcasey/markdown4j) as markdown parsers.
+Apache Zeppelin uses [flexmark](https://github.com/vsch/flexmark-java), [pegdown](https://github.com/sirthias/pegdown) and [markdown4j](https://github.com/jdcasey/markdown4j) as markdown parsers.
In Zeppelin notebook, you can use ` %md ` in the beginning of a paragraph to invoke the Markdown interpreter and generate static html from Markdown plain text.
@@ -53,21 +53,25 @@ For more information, please see [Mathematical Expression](../usage/display_syst
</tr>
<tr>
<td>markdown.parser.type</td>
- <td>pegdown</td>
- <td>Markdown Parser Type. <br/> Available values: pegdown, markdown4j.</td>
+ <td>flexmark</td>
+ <td>Markdown Parser Type. <br/> Available values: flexmark, pegdown, markdown4j.</td>
</tr>
</table>
+### Flexmark parser (Default Markdown Parser)
-### Pegdown Parser
+CommonMark/Markdown Java parser with source level AST.
+
+<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/markdown-example-flexmark-parser.png" width="70%" />
-`pegdown` parser provides github flavored markdown.
+`flexmark` parser provides [YUML](http://yuml.me/) and [Websequence](https://www.websequencediagrams.com/) extensions also.
-<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/markdown-example-pegdown-parser.png" width="70%" />
+<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/markdown-example-flexmark-parser-extensions.png" width="70%" />
-`pegdown` parser provides [YUML](http://yuml.me/) and [Websequence](https://www.websequencediagrams.com/) plugins also.
+### Pegdown Parser
-<img src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/markdown-example-pegdown-parser-plugins.png" width="70%" />
+`pegdown` parser provides github flavored markdown. Although still one of the most popular Markdown parsing libraries for the JVM, pegdown has reached its end of life.
+The project is essentially unmaintained with tickets piling up and crucial bugs not being fixed.`pegdown`'s parsing performance isn't great. But keep this parser for the backward compatibility.
### Markdown4j Parser
diff --git a/markdown/README.md b/markdown/README.md
new file mode 100644
index 0000000..24f5dce
--- /dev/null
+++ b/markdown/README.md
@@ -0,0 +1,34 @@
+# Overview
+Markdown parsers for Apache Zeppelin. Markdown is a plain text formatting syntax designed so that it can be converted to HTML. Apache Zeppelin uses `flexmark`, `pegdown` and `markdown4j`.
+Since both `pegdown` and `markdown4j` are deprecated but it support for backward compatibility.
+
+# Architecture
+Current interpreter implementation creates the instance of parser based on the configuration parameter provided, default is `flexmark` through `Markdown` and render the text into html.
+
+### Technical overview
+When interpreter is starting it check which parser has configured in `markdown.parser.type`, based on this value, it creates the instance of that parser. For each subsequent request,
+respective parser get the markdown text, parses it and renders into the html.
+
+### Flexmark Parser Overview
+
+CommonMark/Markdown Java parser with source level AST.
+
+### Flexmark Requirement
+
+ * maven dependency to add in pom.xml
+
+ ```
+<flexmark.all.version>0.50.40</flexmark.all.version>
+
+ <dependency>
+ <groupId>com.vladsch.flexmark</groupId>
+ <artifactId>flexmark-all</artifactId>
+ <version>${flexmark.all.version}</version>
+ </dependency>
+ ```
+
+### Flexmark Technical overview for Custom Extension
+To support, YUML and websequnce diagram, need to build the image URL from the respective block and render it into HTML, So it requires
+to implement some custom classes. `UMLExtension` is base class which has factory for other classes like `UMLBlockQuoteParser` and `UMLNodeRenderer`.
+`UMLBlockQuoteParser` which parses the UML block and creates block quote node `UMLBlockQuote`.
+`UMLNodeRenderer` which builds the URL using this block quote node `UMLBlockQuote` and render it as image into HTML.
\ No newline at end of file
diff --git a/markdown/pom.xml b/markdown/pom.xml
index 7d6bcff..7720353 100644
--- a/markdown/pom.xml
+++ b/markdown/pom.xml
@@ -38,6 +38,7 @@
<commons.lang3.version>3.4</commons.lang3.version>
<markdown4j.version>2.2-cj-1.0</markdown4j.version>
<pegdown.version>1.6.0</pegdown.version>
+ <flexmark.all.version>0.50.40</flexmark.all.version>
</properties>
<dependencies>
@@ -55,6 +56,12 @@
</dependency>
<dependency>
+ <groupId>com.vladsch.flexmark</groupId>
+ <artifactId>flexmark-all</artifactId>
+ <version>${flexmark.all.version}</version>
+ </dependency>
+
+ <dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>${commons.lang3.version}</version>
diff --git a/markdown/src/main/java/org/apache/zeppelin/markdown/FlexmarkParser.java b/markdown/src/main/java/org/apache/zeppelin/markdown/FlexmarkParser.java
new file mode 100644
index 0000000..f4de944
--- /dev/null
+++ b/markdown/src/main/java/org/apache/zeppelin/markdown/FlexmarkParser.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.markdown;
+
+
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.ext.gfm.strikethrough.StrikethroughExtension;
+import com.vladsch.flexmark.ext.tables.TablesExtension;
+import com.vladsch.flexmark.ext.typographic.TypographicExtension;
+import com.vladsch.flexmark.ext.wikilink.WikiLinkExtension;
+import com.vladsch.flexmark.util.ast.Node;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+
+/**
+ * Flexmark Parser
+ */
+public class FlexmarkParser implements MarkdownParser {
+ private static final Logger LOGGER = LoggerFactory.getLogger(FlexmarkParser.class);
+ private Parser parser;
+ private HtmlRenderer renderer;
+
+ public FlexmarkParser() {
+ MutableDataSet options = new MutableDataSet();
+ options.set(Parser.EXTENSIONS, Arrays.asList(StrikethroughExtension.create(),
+ TablesExtension.create(),
+ UMLExtension.create(),
+ AutolinkExtension.create(),
+ WikiLinkExtension.create(),
+ TypographicExtension.create()));
+ options.set(HtmlRenderer.SOFT_BREAK, "<br />\n");
+ parser = Parser.builder(options).build();
+ renderer = HtmlRenderer.builder(options).build();
+ }
+
+ @Override
+ public String render(String markdownText) {
+ Node document = parser.parse(markdownText);
+ String html = renderer.render(document);
+ return wrapWithMarkdownClassDiv(html);
+ }
+
+ /**
+ * wrap with markdown class div to styling DOM using css.
+ */
+ public static String wrapWithMarkdownClassDiv(String html) {
+ return new StringBuilder()
+ .append("<div class=\"markdown-body\">\n")
+ .append(html)
+ .append("\n</div>")
+ .toString();
+ }
+}
diff --git a/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java b/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java
index 83b4069..99aaac2 100644
--- a/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java
+++ b/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java
@@ -19,7 +19,6 @@ package org.apache.zeppelin.markdown;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-
import java.util.List;
import java.util.Properties;
@@ -56,12 +55,21 @@ public class Markdown extends Interpreter {
public String toString() {
return PARSER_TYPE_MARKDOWN4J;
}
+ },
+
+ FLEXMARK {
+ @Override
+ public String toString() {
+ return PARSER_TYPE_FLEXMARK;
+ }
}
+
}
public static final String MARKDOWN_PARSER_TYPE = "markdown.parser.type";
public static final String PARSER_TYPE_PEGDOWN = "pegdown";
public static final String PARSER_TYPE_MARKDOWN4J = "markdown4j";
+ public static final String PARSER_TYPE_FLEXMARK = "flexmark";
public Markdown(Properties property) {
super(property);
@@ -72,6 +80,8 @@ public class Markdown extends Interpreter {
if (MarkdownParserType.PEGDOWN.toString().equals(parserType)) {
return new PegdownParser();
+ } else if (MarkdownParserType.FLEXMARK.toString().equals(parserType)) {
+ return new FlexmarkParser();
} else {
// default parser
return new Markdown4jParser();
diff --git a/markdown/src/main/java/org/apache/zeppelin/markdown/UMLBlockQuote.java b/markdown/src/main/java/org/apache/zeppelin/markdown/UMLBlockQuote.java
new file mode 100644
index 0000000..4b07134
--- /dev/null
+++ b/markdown/src/main/java/org/apache/zeppelin/markdown/UMLBlockQuote.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.markdown;
+
+import com.vladsch.flexmark.ast.Paragraph;
+import com.vladsch.flexmark.ast.ParagraphContainer;
+import com.vladsch.flexmark.util.ast.Block;
+import com.vladsch.flexmark.util.ast.BlockContent;
+import com.vladsch.flexmark.util.sequence.BasedSequence;
+
+import java.util.List;
+
+/**
+ * Block quote which acts as a node.
+ */
+public class UMLBlockQuote extends Block implements ParagraphContainer {
+
+ private BasedSequence openingMarker = BasedSequence.NULL;
+ private BasedSequence openingTrailing = BasedSequence.NULL;
+ private BasedSequence closingMarker = BasedSequence.NULL;
+ private BasedSequence closingTrailing = BasedSequence.NULL;
+
+ @Override
+ public void getAstExtra(StringBuilder out) {
+ segmentSpanChars(out, openingMarker, "open");
+ segmentSpanChars(out, openingTrailing, "openTrail");
+ segmentSpanChars(out, closingMarker, "close");
+ segmentSpanChars(out, closingTrailing, "closeTrail");
+ }
+
+ @Override
+ public BasedSequence[] getSegments() {
+ return new BasedSequence[]{openingMarker, openingTrailing, closingMarker, closingTrailing};
+ }
+
+ @Override
+ public boolean isParagraphEndWrappingDisabled(final Paragraph node) {
+ return node == getLastChild() || node.getNext() instanceof UMLBlockQuote;
+ }
+
+ @Override
+ public boolean isParagraphStartWrappingDisabled(final Paragraph node) {
+ return node == getFirstChild() || node.getPrevious() instanceof UMLBlockQuote;
+ }
+
+ public UMLBlockQuote() {
+ }
+
+ public UMLBlockQuote(BasedSequence chars) {
+ super(chars);
+ }
+
+ public UMLBlockQuote(BasedSequence chars, List<BasedSequence> segments) {
+ super(chars, segments);
+ }
+
+ public UMLBlockQuote(BlockContent blockContent) {
+ super(blockContent);
+ }
+
+ public BasedSequence getOpeningMarker() {
+ return openingMarker;
+ }
+
+ public void setOpeningMarker(BasedSequence openingMarker) {
+ this.openingMarker = openingMarker;
+ }
+
+ public BasedSequence getClosingMarker() {
+ return closingMarker;
+ }
+
+ public void setClosingMarker(final BasedSequence closingMarker) {
+ this.closingMarker = closingMarker;
+ }
+
+ public BasedSequence getOpeningTrailing() {
+ return openingTrailing;
+ }
+
+ public void setOpeningTrailing(final BasedSequence openingTrailing) {
+ this.openingTrailing = openingTrailing;
+ }
+
+ public BasedSequence getClosingTrailing() {
+ return closingTrailing;
+ }
+
+ public void setClosingTrailing(final BasedSequence closingTrailing) {
+ this.closingTrailing = closingTrailing;
+ }
+
+ @Override
+ public String toString() {
+ return "YUMLBlockQuote{" +
+ "openingMarker=" + openingMarker +
+ ", openingTrailing=" + openingTrailing +
+ ", closingMarker=" + closingMarker +
+ ", closingTrailing=" + closingTrailing +
+ ", lineSegments=" + lineSegments +
+ '}';
+ }
+}
diff --git a/markdown/src/main/java/org/apache/zeppelin/markdown/UMLBlockQuoteParser.java b/markdown/src/main/java/org/apache/zeppelin/markdown/UMLBlockQuoteParser.java
new file mode 100644
index 0000000..9d56856
--- /dev/null
+++ b/markdown/src/main/java/org/apache/zeppelin/markdown/UMLBlockQuoteParser.java
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.markdown;
+
+import com.vladsch.flexmark.ext.gitlab.internal.GitLabOptions;
+import com.vladsch.flexmark.parser.InlineParser;
+import com.vladsch.flexmark.parser.block.AbstractBlockParser;
+import com.vladsch.flexmark.parser.block.BlockContinue;
+import com.vladsch.flexmark.parser.block.BlockParser;
+import com.vladsch.flexmark.parser.block.BlockParserFactory;
+import com.vladsch.flexmark.parser.block.AbstractBlockParserFactory;
+import com.vladsch.flexmark.parser.block.BlockStart;
+import com.vladsch.flexmark.parser.block.MatchedBlockParser;
+import com.vladsch.flexmark.parser.block.ParserState;
+import com.vladsch.flexmark.parser.block.CustomBlockParserFactory;
+import com.vladsch.flexmark.util.ast.Block;
+import com.vladsch.flexmark.util.ast.BlockContent;
+import com.vladsch.flexmark.util.ast.Node;
+import com.vladsch.flexmark.util.data.DataHolder;
+import com.vladsch.flexmark.util.sequence.BasedSequence;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Parser to get the Block content
+ */
+public class UMLBlockQuoteParser extends AbstractBlockParser {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UMLBlockQuoteParser.class);
+
+
+ private static Pattern YUML_BLOCK_START = Pattern.compile("(%%%)\\s+(.*\\n)");
+ private static Pattern YUML_BLOCK_END = Pattern.compile("%%%(\\s*$)");
+
+ private final UMLBlockQuote block = new UMLBlockQuote();
+ private BlockContent content = new BlockContent();
+ private final GitLabOptions options;
+ private boolean hadClose = false;
+
+ UMLBlockQuoteParser(DataHolder options, BasedSequence openMarker, BasedSequence openTrailing) {
+ this.options = new GitLabOptions(options);
+ this.block.setOpeningMarker(openMarker);
+ this.block.setOpeningTrailing(openTrailing);
+ }
+
+ @Override
+ public Block getBlock() {
+ return block;
+ }
+
+ @Override
+ public BlockContinue tryContinue(ParserState state) {
+ if (hadClose) {
+ return BlockContinue.none();
+ }
+
+ final int index = state.getIndex();
+
+ BasedSequence line = state.getLineWithEOL();
+ final Matcher matcher = YUML_BLOCK_END.matcher(line.subSequence(index));
+ if (!matcher.matches()) {
+ return BlockContinue.atIndex(index);
+ } else {
+ // if have open gitlab block quote last child then let them handle it
+ Node lastChild = block.getLastChild();
+ if (lastChild instanceof UMLBlockQuote) {
+ final BlockParser parser = state.getActiveBlockParser((Block) lastChild);
+ if (parser instanceof UMLBlockQuoteParser && !((UMLBlockQuoteParser) parser).hadClose) {
+ // let the child handle it
+ return BlockContinue.atIndex(index);
+ }
+ }
+ hadClose = true;
+ block.setClosingMarker(state.getLine().subSequence(index, index + 3));
+ block.setClosingTrailing(state.getLineWithEOL().subSequence(matcher.start(1),
+ matcher.end(1)));
+ return BlockContinue.atIndex(state.getLineEndIndex());
+ }
+ }
+
+ @Override
+ public void addLine(ParserState state, BasedSequence line) {
+ content.add(line, state.getIndent());
+ }
+
+ @Override
+ public void closeBlock(ParserState state) {
+ block.setContent(content);
+ block.setCharsFromContent();
+ content = null;
+ }
+
+ @Override
+ public boolean isContainer() {
+ return true;
+ }
+
+ @Override
+ public boolean canContain(final ParserState state, final BlockParser blockParser,
+ final Block block) {
+ return true;
+ }
+
+ @Override
+ public void parseInlines(InlineParser inlineParser) {
+ }
+
+ /**
+ * Generic Factory
+ */
+ public static class Factory implements CustomBlockParserFactory {
+ @Override
+ public Set<Class<? extends CustomBlockParserFactory>> getAfterDependents() {
+ return null;
+ }
+
+ @Override
+ public Set<Class<? extends CustomBlockParserFactory>> getBeforeDependents() {
+ return null;
+ }
+
+ @Override
+ public boolean affectsGlobalScope() {
+ return false;
+ }
+
+ @Override
+ public BlockParserFactory apply(DataHolder options) {
+ return new UMLBlockQuoteParser.BlockFactory(options);
+ }
+ }
+
+ private static class BlockFactory extends AbstractBlockParserFactory {
+ private final GitLabOptions options;
+
+ BlockFactory(DataHolder options) {
+ super(options);
+ this.options = new GitLabOptions(options);
+ }
+
+ boolean haveBlockQuoteParser(ParserState state) {
+ final List<BlockParser> parsers = state.getActiveBlockParsers();
+ int i = parsers.size();
+ while (i-- > 0) {
+ if (parsers.get(i) instanceof UMLBlockQuoteParser) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public BlockStart tryStart(ParserState state, MatchedBlockParser matchedBlockParser) {
+ if (options.nestedBlockQuotes || !haveBlockQuoteParser(state)) {
+ BasedSequence line = state.getLineWithEOL();
+ final Matcher matcher = YUML_BLOCK_START.matcher(line);
+ if (matcher.matches()) {
+ LOGGER.debug("Matcher group count {} ", matcher.groupCount());
+ return BlockStart.of(new UMLBlockQuoteParser(state.getProperties(),
+ line.subSequence(0, 3), line.subSequence(4, line.length())))
+ .atIndex(state.getLineEndIndex());
+ }
+ }
+ return BlockStart.none();
+ }
+ }
+}
diff --git a/markdown/src/main/java/org/apache/zeppelin/markdown/UMLExtension.java b/markdown/src/main/java/org/apache/zeppelin/markdown/UMLExtension.java
new file mode 100644
index 0000000..4956a31
--- /dev/null
+++ b/markdown/src/main/java/org/apache/zeppelin/markdown/UMLExtension.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.markdown;
+
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.util.data.MutableDataHolder;
+import com.vladsch.flexmark.util.builder.Extension;
+
+
+/**
+ * Extension to support YUML and web sequnce diagram.
+ */
+public class UMLExtension implements Parser.ParserExtension, HtmlRenderer.HtmlRendererExtension {
+
+ private UMLExtension() {
+
+ }
+
+ public static Extension create() {
+ return new UMLExtension();
+ }
+
+ @Override
+ public void rendererOptions(MutableDataHolder options) {
+
+ }
+
+ @Override
+ public void extend(HtmlRenderer.Builder rendererBuilder, String rendererType) {
+ rendererBuilder.nodeRendererFactory(new UMLNodeRenderer.Factory());
+ }
+
+ @Override
+ public void parserOptions(MutableDataHolder options) {
+ }
+
+ @Override
+ public void extend(Parser.Builder parserBuilder) {
+ parserBuilder.customBlockParserFactory(new UMLBlockQuoteParser.Factory());
+ }
+}
diff --git a/markdown/src/main/java/org/apache/zeppelin/markdown/UMLNodeRenderer.java b/markdown/src/main/java/org/apache/zeppelin/markdown/UMLNodeRenderer.java
new file mode 100644
index 0000000..39406f9
--- /dev/null
+++ b/markdown/src/main/java/org/apache/zeppelin/markdown/UMLNodeRenderer.java
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.markdown;
+
+import com.vladsch.flexmark.ext.gitlab.internal.GitLabOptions;
+import com.vladsch.flexmark.html.CustomNodeRenderer;
+import com.vladsch.flexmark.html.HtmlWriter;
+import com.vladsch.flexmark.html.renderer.NodeRenderer;
+import com.vladsch.flexmark.html.renderer.NodeRendererFactory;
+import com.vladsch.flexmark.html.renderer.NodeRendererContext;
+import com.vladsch.flexmark.html.renderer.NodeRenderingHandler;
+import com.vladsch.flexmark.util.data.DataHolder;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.io.UnsupportedEncodingException;
+import java.net.URL;
+import java.net.URLConnection;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.HashMap;
+import java.util.Objects;
+
+import static org.apache.commons.lang3.StringUtils.defaultString;
+
+/**
+ * Html Node renderer to render the image
+ */
+public class UMLNodeRenderer implements NodeRenderer {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(UMLNodeRenderer.class);
+
+ public static final String YUML = "yuml";
+ public static final String SEQUENCE = "sequence";
+ public static final String WEBSEQ_URL = "http://www.websequencediagrams.com";
+
+ final GitLabOptions options;
+
+ public UMLNodeRenderer(DataHolder options) {
+ this.options = new GitLabOptions(options);
+ }
+
+ @Override
+ public Set<NodeRenderingHandler<?>> getNodeRenderingHandlers() {
+ Set<NodeRenderingHandler<?>> set = new HashSet<>();
+ set.add(new NodeRenderingHandler<>(UMLBlockQuote.class,
+ new CustomNodeRenderer<UMLBlockQuote>() {
+ @Override
+ public void render(UMLBlockQuote node, NodeRendererContext context,
+ HtmlWriter html) {
+ UMLNodeRenderer.this.render(node, context, html);
+ }
+ }));
+ return set;
+ }
+
+
+ private void render(final UMLBlockQuote node, final NodeRendererContext context,
+ HtmlWriter html) {
+ LOGGER.debug("Rendering HTML");
+
+ String firstLine = node.getOpeningTrailing().toString();
+ String[] splitWithSpace = firstLine.split(" ");
+
+ LOGGER.debug("Start of the node {} ", firstLine);
+ LOGGER.debug("Content within block {} ", node.getFirstChild().getChars());
+
+ Map<String, String> paramMap = new HashMap<>();
+ for (int i = 1; i < splitWithSpace.length; i++) {
+ String[] splitWithEqual = splitWithSpace[i].split("=");
+ paramMap.put(splitWithEqual[0], splitWithEqual[1]);
+ }
+
+ String url = "";
+
+ if (splitWithSpace[0].equals(YUML) && !Objects.isNull(node.getFirstChild())) {
+ url = createYumlUrl(paramMap, node.getFirstChild().getChars().toString());
+ LOGGER.debug("Encoded YUML URL {} ", url);
+ } else if (splitWithSpace[0].equals(SEQUENCE) && !Objects.isNull(node.getFirstChild())) {
+ url = createWebsequenceUrl(paramMap.get("style"), node.getFirstChild().getChars().toString());
+ LOGGER.debug("Encoded web sequence diagram URL {} ", url);
+ } else {
+ html.withAttr().tagLineIndent("blockquote", new Runnable() {
+ @Override
+ public void run() {
+ context.renderChildren(node);
+ }
+ });
+ return;
+ }
+
+ html.attr("src", url);
+ html.attr("alt", "");
+ html.srcPos(node.getChars()).withAttr().tagVoid("img");
+ }
+
+ /**
+ * Factory for node renderer
+ */
+ public static class Factory implements NodeRendererFactory {
+ @Override
+ public NodeRenderer apply(final DataHolder options) {
+ return new UMLNodeRenderer(options);
+ }
+ }
+
+ public static String createYumlUrl(Map<String, String> params, String body) {
+ StringBuilder inlined = new StringBuilder();
+ for (String line : body.split("\\r?\\n")) {
+ line = line.trim();
+ if (line.length() > 0) {
+ if (inlined.length() > 0) {
+ inlined.append(", ");
+ }
+ inlined.append(line);
+ }
+ }
+
+ String encodedBody = null;
+ try {
+ encodedBody = URLEncoder.encode(inlined.toString(), "UTF-8");
+ } catch (UnsupportedEncodingException e) {
+ new RuntimeException("Failed to encode YUML markdown body", e);
+ }
+
+ StringBuilder mergedStyle = new StringBuilder();
+ String style = defaultString(params.get("style"), "scruffy");
+ String type = defaultString(params.get("type"), "class");
+ String format = defaultString(params.get("format"), "svg");
+
+ mergedStyle.append(style);
+
+ if (null != params.get("dir")) {
+ mergedStyle.append(";dir:" + params.get("dir"));
+ }
+
+ if (null != params.get("scale")) {
+ mergedStyle.append(";scale:" + params.get("scale"));
+ }
+
+ return new StringBuilder()
+ .append("http://yuml.me/diagram/")
+ .append(mergedStyle.toString() + "/")
+ .append(type + "/")
+ .append(encodedBody)
+ .append("." + format)
+ .toString();
+ }
+
+ public static String createWebsequenceUrl(String style,
+ String content) {
+
+ style = StringUtils.defaultString(style, "default");
+
+ OutputStreamWriter writer = null;
+ BufferedReader reader = null;
+
+ String webSeqUrl = "";
+
+ try {
+ String query = new StringBuilder()
+ .append("style=")
+ .append(style)
+ .append("&message=")
+ .append(URLEncoder.encode(content, "UTF-8"))
+ .append("&apiVersion=1")
+ .toString();
+
+ URL url = new URL(WEBSEQ_URL);
+ URLConnection conn = url.openConnection();
+ conn.setDoOutput(true);
+ writer = new OutputStreamWriter(conn.getOutputStream(), StandardCharsets.UTF_8);
+ writer.write(query);
+ writer.flush();
+
+ StringBuilder response = new StringBuilder();
+ reader = new BufferedReader(
+ new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8));
+ String line;
+ while ((line = reader.readLine()) != null) {
+ response.append(line);
+ }
+
+ writer.close();
+ reader.close();
+
+ String json = response.toString();
+
+ int start = json.indexOf("?png=");
+ int end = json.indexOf("\"", start);
+
+ if (start != -1 && end != -1) {
+ webSeqUrl = WEBSEQ_URL + "/" + json.substring(start, end);
+ System.out.println("websequrl : " + webSeqUrl);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to get proper response from websequencediagrams.com", e);
+ } finally {
+ IOUtils.closeQuietly(writer);
+ IOUtils.closeQuietly(reader);
+ }
+
+ return webSeqUrl;
+ }
+}
diff --git a/markdown/src/main/resources/interpreter-setting.json b/markdown/src/main/resources/interpreter-setting.json
index d2a59c8..da4b59b 100644
--- a/markdown/src/main/resources/interpreter-setting.json
+++ b/markdown/src/main/resources/interpreter-setting.json
@@ -7,8 +7,8 @@
"markdown.parser.type": {
"envName": "MARKDOWN_PARSER_TYPE",
"propertyName": "markdown.parser.type",
- "defaultValue": "pegdown",
- "description": "Markdown Parser Type. Available values: pegdown, markdown4j. Default = pegdown",
+ "defaultValue": "flexmark",
+ "description": "Markdown Parser Type. Available values: pegdown, markdown4j, flexmark. Default = flexmark",
"type": "string"
}
},
diff --git a/markdown/src/test/java/org/apache/zeppelin/markdown/FlexmarkParserTest.java b/markdown/src/test/java/org/apache/zeppelin/markdown/FlexmarkParserTest.java
new file mode 100644
index 0000000..a80337e
--- /dev/null
+++ b/markdown/src/test/java/org/apache/zeppelin/markdown/FlexmarkParserTest.java
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.markdown;
+
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ErrorCollector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Properties;
+
+import static org.apache.zeppelin.markdown.FlexmarkParser.wrapWithMarkdownClassDiv;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+public class FlexmarkParserTest {
+
+ Logger logger = LoggerFactory.getLogger(FlexmarkParserTest.class);
+ Markdown md;
+
+ @Rule
+ public ErrorCollector collector = new ErrorCollector();
+
+ @Before
+ public void setUp() throws Exception {
+ Properties props = new Properties();
+ props.put(Markdown.MARKDOWN_PARSER_TYPE, Markdown.PARSER_TYPE_FLEXMARK);
+ md = new Markdown(props);
+ md.open();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ md.close();
+ }
+
+ @Test
+ public void testMultipleThread() {
+ ArrayList<Thread> arrThreads = new ArrayList<Thread>();
+ for (int i = 0; i < 10; i++) {
+ Thread t = new Thread() {
+ public void run() {
+ String r1 = null;
+ try {
+ r1 = md.interpret("# H1", null).code().name();
+ } catch (Exception e) {
+ logger.error("testTestMultipleThread failed to interpret", e);
+ }
+ collector.checkThat("SUCCESS",
+ CoreMatchers.containsString(r1));
+ }
+ };
+ t.start();
+ arrThreads.add(t);
+ }
+
+ for (int i = 0; i < 10; i++) {
+ try {
+ arrThreads.get(i).join();
+ } catch (InterruptedException e) {
+ logger.error("testTestMultipleThread failed to join threads", e);
+ }
+ }
+ }
+
+ @Test
+ public void testStrikethrough() {
+ InterpreterResult result = md.interpret("This is ~~deleted~~ text", null);
+ assertEquals(wrapWithMarkdownClassDiv("<p>This is <del>deleted</del> text</p>\n"),
+ result.message().get(0).getData());
+ }
+
+ @Test
+ public void testHeader() {
+ InterpreterResult r1 = md.interpret("# H1", null);
+ assertEquals(wrapWithMarkdownClassDiv("<h1>H1</h1>\n"), r1.message().get(0).getData());
+
+ InterpreterResult r2 = md.interpret("## H2", null);
+ assertEquals(wrapWithMarkdownClassDiv("<h2>H2</h2>\n"), r2.message().get(0).getData());
+
+ InterpreterResult r3 = md.interpret("### H3", null);
+ assertEquals(wrapWithMarkdownClassDiv("<h3>H3</h3>\n"), r3.message().get(0).getData());
+
+ InterpreterResult r4 = md.interpret("#### H4", null);
+ assertEquals(wrapWithMarkdownClassDiv("<h4>H4</h4>\n"), r4.message().get(0).getData());
+
+ InterpreterResult r5 = md.interpret("##### H5", null);
+ assertEquals(wrapWithMarkdownClassDiv("<h5>H5</h5>\n"), r5.message().get(0).getData());
+
+ InterpreterResult r6 = md.interpret("###### H6", null);
+ assertEquals(wrapWithMarkdownClassDiv("<h6>H6</h6>\n"), r6.message().get(0).getData());
+
+ InterpreterResult r7 = md.interpret("Alt-H1\n" + "======", null);
+ assertEquals(wrapWithMarkdownClassDiv("<h1>Alt-H1</h1>\n"), r7.message().get(0).getData());
+
+ InterpreterResult r8 = md.interpret("Alt-H2\n" + "------", null);
+ assertEquals(wrapWithMarkdownClassDiv("<h2>Alt-H2</h2>\n"), r8.message().get(0).getData());
+ }
+
+ @Test
+ public void testItalics() {
+ InterpreterResult result = md.interpret("This is *italics* text", null);
+
+ assertEquals(
+ wrapWithMarkdownClassDiv("<p>This is <em>italics</em> text</p>\n"),
+ result.message().get(0).getData());
+ }
+
+ @Test
+ public void testStrongEmphasis() {
+ InterpreterResult result = md.interpret("This is **strong emphasis** text", null);
+ assertEquals(
+ wrapWithMarkdownClassDiv("<p>This is <strong>strong emphasis</strong> text</p>\n"),
+ result.message().get(0).getData());
+ }
+
+ @Test
+ public void testOrderedList() {
+ String input =
+ new StringBuilder()
+ .append("1. First ordered list item\n")
+ .append("2. Another item")
+ .toString();
+
+ String expected =
+ new StringBuilder()
+ .append("<ol>\n")
+ .append("<li>First ordered list item</li>\n")
+ .append("<li>Another item</li>\n")
+ .append("</ol>\n")
+ .toString();
+
+ InterpreterResult result = md.interpret(input, null);
+
+
+ assertEquals(wrapWithMarkdownClassDiv(expected), result.message().get(0).getData());
+ }
+
+ @Test
+ public void testUnorderedList() {
+ String input =
+ new StringBuilder()
+ .append("* Unordered list can use asterisks\n")
+ .append("- Or minuses\n")
+ .append("+ Or pluses")
+ .toString();
+
+ String expected =
+ new StringBuilder()
+ .append("<ul>\n")
+ .append("<li>Unordered list can use asterisks</li>\n")
+ .append("</ul>\n")
+ .append("<ul>\n")
+ .append("<li>Or minuses</li>\n")
+ .append("</ul>\n")
+ .append("<ul>\n")
+ .append("<li>Or pluses</li>\n")
+ .append("</ul>\n")
+ .toString();
+
+ InterpreterResult result = md.interpret(input, null);
+
+ assertEquals(wrapWithMarkdownClassDiv(expected), result.message().get(0).getData());
+ }
+
+ @Test
+ public void testYumlPlugin() {
+ String input = new StringBuilder()
+ .append("%%% yuml style=nofunky scale=120 format=svg\n")
+ .append("[Customer]<>-orders>[Order]\n")
+ .append("[Order]++-0..>[LineItem]\n")
+ .append("[Order]-[note:Aggregate root.]\n")
+ .append(" %%% ")
+ .toString();
+
+ InterpreterResult result = md.interpret(input, null);
+ assertThat(result.message().get(0).getData(), CoreMatchers
+ .containsString("<img src=\"http://yuml.me/diagram/"));
+ }
+
+ @Test
+ public void testWebsequencePlugin() {
+ String input =
+ new StringBuilder()
+ .append("%%% sequence style=modern-blue\n")
+ .append("title Authentication Sequence\n")
+ .append("Alice->Bob: Authentication Request\n")
+ .append("note right of Bob: Bob thinks about it\n")
+ .append("Bob->Alice: Authentication Response\n")
+ .append(" %%% ")
+ .toString();
+
+ InterpreterResult result = md.interpret(input, null);
+
+ System.err.println(result.message().get(0).getData());
+ if (!result.message().get(0).getData().contains(
+ "<img src=\"http://www.websequencediagrams.com/?png=")) {
+ logger.error("Expected {} but found {}",
+ "<img src=\"http://www.websequencediagrams.com/?png=", result.message().get(0).getData());
+ }
+ }
+
+}
+