You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by "Cole-Greer (via GitHub)" <gi...@apache.org> on 2026/05/03 05:36:10 UTC
[PR] Replace AWK/shell docs preprocessing with AsciidoctorJ extension [tinkerpop]
Cole-Greer opened a new pull request, #3418:
URL: https://github.com/apache/tinkerpop/pull/3418
## Replace AWK/shell docs preprocessing with AsciidoctorJ extension
### Overview
This PR introduces the `gremlin-docs` module — a custom AsciidoctorJ TreeProcessor extension that replaces the entire AWK/shell preprocessing pipeline used to build TinkerPop documentation. The old system consisted of 10 AWK scripts, 5 shell scripts, and required a running Gremlin Server, Hadoop daemons, and a built Gremlin Console distribution. The new system runs entirely within the Maven build via an embedded `GremlinGroovyScriptEngine`.
### What it does
- **Executes `[gremlin-groovy]` code blocks** in an embedded script engine during Asciidoctor rendering, producing live console output with `gremlin>` prompts and `==>` results
- **Auto-generates language variant tabs** (Java, Python, JavaScript, C#, Go) for every translatable Gremlin example using the ANTLR-based `GremlinTranslator`
- **Supports Hadoop/Spark examples** via local-mode Spark with a sandboxed filesystem for `hdfs` operations
- **Handles standalone tab groups** (`[source,LANG,tab]` blocks) for manually-authored multi-language examples
- **Detects console-only blocks** (`:remote`, `:>`, `:submit`) and renders them as static code
- **Loads GremlinPlugin customizers** via SPI (hadoop-gremlin, spark-gremlin, gremlin-console) for imports, bindings, and utility functions like `describeGraph()`
### How to use
```bash
bin/process-docs-new.sh # full build with live gremlin execution
bin/process-docs-new.sh --dry-run # skip execution (fast, for layout checks)
```
### Changes
- New `gremlin-docs/` module (not in the Maven reactor — built separately by the build script)
- New `bin/process-docs-new.sh` build entry point
- Root `pom.xml`: added `gremlin-docs` as `asciidoctor-maven-plugin` dependency, switched syntax highlighter from CodeRay to highlight.js 11.9.0, added `tabs-1` CSS rule
- `docs/stylesheets/tinkerpop.css`: added `.tabs-1` CSS for single-tab blocks
### Known Issues
**Callout numbers not rendering in code blocks.** The circled callout numbers (①②③) that appear inline in code examples are not visible. The `<b class="conum">` elements are present in the HTML but highlight.js destroys them when it processes the `<code>` block innerHTML. The callout explanation lists below the code blocks render correctly — only the inline markers are affected.
**Path and other datatype formatting differs from console output.** The Gremlin Console applies custom string conversions for certain datatypes (e.g., `Path` renders as `[v[1],v[2],vadas]` in the console but `path[v[1], v[2], vadas]` in the embedded engine). This affects the displayed results but not their correctness. Other datatypes with console-specific formatting may also differ.
**SPARQL examples fail (~18 errors).** The `sparql-gremlin` plugin's `SparqlTraversalSource` and `g.sparql()` calls fail because the SPARQL plugin is not fully loaded in the embedded engine. These blocks fall back to dry-run output showing `gremlin>` prompts without results.
**Neo4j examples fail (~6 errors).** Blocks using `Neo4jGraph`, `graph.cypher()`, `vertex.addLabel()`, and other Neo4j-specific APIs fail because Neo4j is not on the classpath. These fall back to dry-run output.
**Sugar plugin examples fail (~8 errors).** Groovy sugar syntax (`g.V.name`, `g.V[0..2]`, `g.V.outE.weight`) fails because the Sugar plugin's AST transformations are not active in the embedded engine.
**Remote connection examples fail (~10 errors).** Blocks using `Cluster.open()`, `DriverRemoteConnection`, and `traversal().with('conf/remote-graph.properties')` fail because there is no running Gremlin Server. These are detected and rendered as static code when they contain `:remote`/`:>` commands, but some blocks use the driver API directly without console commands.
**A few Hadoop `hdfs.head()` calls fail.** The `hdfs.head('output/~g')` and `hdfs.head('output', GryoInputFormat)` calls fail because the Spark computation output format differs slightly in local mode vs a real cluster.
**No `x.y.z` version replacement in tab content.** The version placeholder `x.y.z` is replaced in the post-processing step via sed, but content inside tabs generated by the extension may not be caught if the placeholder appears in translated code.
### Not changed
- The old `bin/process-docs.sh` and AWK pipeline are untouched — both systems can coexist
- All AsciiDoc source files are unchanged
- Document structure, sections, anchors, and static content are preserved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Replace AWK/shell docs preprocessing with AsciidoctorJ extension [tinkerpop]
Posted by "spmallette (via GitHub)" <gi...@apache.org>.
spmallette commented on code in PR #3418:
URL: https://github.com/apache/tinkerpop/pull/3418#discussion_r3181689685
##########
gremlin-docs/pom.xml:
##########
@@ -0,0 +1,72 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
Review Comment:
If this is a standalone utility/tool, should it use TinkerPop as a parent? Could we tuck gremlin-docs away in the `docs/` directory? Perhaps even better would be to call it "tinkerpop-docs" to not confuse it with our standard "gremlin" prefixing/suffixing?
bigger thinking - i think there's going to be more and more of these types of little tools and things for various bits of automation and such. not sure where those things should live. i think `docs/upgrade` was one of those to start it. maybe the examples too. they all sorta build outside of the primary maven reactor. Do we need something else that ties these together better? almost need a top-level command system of sorts to keep all these tools and scripts unified somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Replace AWK/shell docs preprocessing with AsciidoctorJ extension [tinkerpop]
Posted by "Cole-Greer (via GitHub)" <gi...@apache.org>.
Cole-Greer closed pull request #3418: Replace AWK/shell docs preprocessing with AsciidoctorJ extension
URL: https://github.com/apache/tinkerpop/pull/3418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Replace AWK/shell docs preprocessing with AsciidoctorJ extension [tinkerpop]
Posted by "Cole-Greer (via GitHub)" <gi...@apache.org>.
Cole-Greer commented on PR #3418:
URL: https://github.com/apache/tinkerpop/pull/3418#issuecomment-4662472921
Superseded by https://github.com/apache/tinkerpop/pull/3455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tinkerpop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org