You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "mbeckerle (via GitHub)" <gi...@apache.org> on 2025/10/24 20:59:24 UTC
[PR] Updating documentation on site - PDF creation for some files [daffodil-site]
mbeckerle opened a new pull request, #193:
URL: https://github.com/apache/daffodil-site/pull/193
This is a work in progress preview.
Please run the site locally and look at the dfdl-extensions page and the layers page, and click the link to show the PDF files (near top of page).
This contains pandoc so a common markdown file
with some conditionalizations in it
can generate both web pages using
jekyll as usual, but also
pdf files from the same markdown.
Currently this requires you to cd to the site/_pandoc/ directory and type 'make' to run the makefile.
That will find all pages with 'pdf: true' in their header
and use pandoc to create PDFs for them.
The PDFs are placed in site/pdf, and are currently checked into the repo.
A drawback is you have to use a common markdown subset that works in jekyll and pandoc.
And you have to remember to 'make' the PDFs.
This is intended for a subset of our documentation to use. Not every page has to have a PDF.
DAFFODIL-3044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updating documentation on site - PDF creation for some files [daffodil-site]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #193:
URL: https://github.com/apache/daffodil-site/pull/193#discussion_r2469791351
##########
site/_pandoc/list-pdf-sources.awk:
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/awk -f
+# Prints FILENAME iff the file has YAML front matter with "pdf: true".
+# - Must be called with filenames (works via find/xargs).
+# - Ignores matches outside the front matter.
+# - Front matter is the lines between the first '---' and the next '---'.
+
+# We process each file independently.
+# Use a per-file BEGINFILE block if available (GNU awk). Otherwise reset on first record.
+BEGIN {
+ have_beginfile = 0
+}
+BEGINFILE {
+ have_beginfile = 1
+ in_front = 0
+ seen_start = 0
+ want_pdf = 0
+}
+
+# For non-GNU awk compatibility:
+# Reset on the first line of each file if BEGINFILE isn't supported.
+FNR == 1 && !have_beginfile {
+ in_front = 0
+ seen_start = 0
+ want_pdf = 0
+}
+
+{
+ # Detect start of front matter
+ if (!seen_start) {
+ if ($0 ~ /^[[:space:]]*---[[:space:]]*$/) {
+ seen_start = 1
+ in_front = 1
+ next
+ } else {
+ # No front matter: skip file
+ nextfile
+ }
+ }
+
+ # If in front matter, look for end and for pdf:true
+ if (in_front) {
+ if ($0 ~ /^[[:space:]]*---[[:space:]]*$/) {
+ # End of front matter
+ in_front = 0
+ if (want_pdf) {
+ print FILENAME
+ }
+ nextfile
+ }
+ # Match "pdf: true" allowing spaces; ensure it's a key at line start
+ if ($0 ~ /^[[:space:]]*pdf:[[:space:]]*true([[:space:]]|$)/) {
+ want_pdf = 1
+ }
+ next
+ }
+
+ # If we got here, weโve passed front matter without finding pdf:true
+ nextfile
+}
Review Comment:
One-liner grep in prior comment supercedes any need for this script as awk or bash.
##########
site/_pandoc/Makefile:
##########
@@ -0,0 +1,63 @@
+# ==========================================================
+# Pandoc PDF generator for Jekyll site
+# Scans Markdown files with "pdf: true" in YAML front matter
+# and produces PDFs in the site's ./pdf/ directory
+# ==========================================================
+
+# --- Configuration ---
+SITE_ROOT := ..
+AWK_UNWRAP := $(SITE_ROOT)/_pandoc/unwrap-pandoc.awk
+AWK_LIST := $(SITE_ROOT)/_pandoc/list-pdf-sources.awk
+PANDOC := pandoc
+
+# Output directory for generated PDFs (at site root)
+PDF_OUTDIR := $(SITE_ROOT)/pdf
+
+DEFAULTS := $(SITE_ROOT)/_pandoc/basic.yaml
+
+# --- Candidate Markdown files (exclude build/tool/output dirs) ---
+# Use find + awk pipeline โ awk -f avoids executable bit.
+MD_CANDIDATES := $(shell find $(SITE_ROOT) \
+ -type f -name '*.md' \
+ -not -path '*/_*/*' \
+ -not -path '*/node_modules/*' \
+ -not -path '*/vendor/*' \
+ -not -path '*/pdf/*' \
+ -print0 | xargs -0 -r awk -f $(AWK_LIST))
Review Comment:
Fixed
##########
site/dfdl-extensions.md:
##########
@@ -21,38 +22,60 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
+<!--
+The :target="_blank" syntax below makes this open in a new tab
+and work in the PDF and jekyll web page.
+But displays as literal text in the IDE markdown previewer.
+-->
+<div class="only-jekyll" markdown="1">
+_This page is available as a [downloadable PDF](../pdf/dfdl-extensions.pdf){:target="_blank"}._
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+</div>
+
+<div class="only-pandoc" markdown="1">
+# Introduction
+</div>
Review Comment:
Done.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
Review Comment:
Updated
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
Review Comment:
fixed
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
+```
+
+keeping them separate from the Jekyll site itself.
+
+---
+
+## ๐งฎ Example Commands
+
+From inside the `_pandoc/` directory:
+
+### Build all PDFs
+```bash
+make
+```
+
+### Clean all generated PDFs
+```bash
+make clean
+```
+
+### List all Markdown files with `pdf: true`
+```bash
+make list
+```
+
+### Force rebuild of one PDF
+```bash
+make ../about.pdf
+```
+
+---
+
+## ๐งฐ How the AWK Script Works
+
+`unwrap-pandoc.awk` removes the HTML comment wrappers used to hide LaTeX from Jekyll.
+
+Input example:
+
+````markdown
+<!-- PANDOC:START -->
+<!--
+\LaTeX code
+-->
+<!-- PANDOC:END -->
+````
+
+Output to Pandoc:
+
+```markdown
+\LaTeX code
+```
+
+That means Pandoc receives clean, valid LaTeX syntax while Jekyll never sees it.
+
+---
Review Comment:
removed section
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
Review Comment:
fixed
##########
site/dfdl-extensions.md:
##########
@@ -21,38 +22,60 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
+<!--
+The :target="_blank" syntax below makes this open in a new tab
+and work in the PDF and jekyll web page.
+But displays as literal text in the IDE markdown previewer.
+-->
+<div class="only-jekyll" markdown="1">
+_This page is available as a [downloadable PDF](../pdf/dfdl-extensions.pdf){:target="_blank"}._
Review Comment:
Done. Had to put it into default.html and use page.path etc. but works. However, pages still have conditionalization about the jekyll table of contents. That is, if you want the jekyll page to have a TOC, you have to include the only-jekyll conditional TOC boilerplate at the top (because pandoc generates the TOC without needing this, and ends up including the jekyll code as plain text). I think PDF pages are probably going to want a TOC usually both on the web and PDF, but I haven't figured out how to centralize this yet.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
Review Comment:
removed irrelevant parts.
##########
site/_pandoc/only.lua:
##########
@@ -0,0 +1,60 @@
+-- only.lua: drop .only-jekyll, keep contents of .only-pandoc
+-- Handles both native Div/Span nodes and raw HTML <div> wrappers.
+
+local List = require 'pandoc.List'
+
+local function has_class(classes, cls)
+ return classes and List.includes(classes, cls)
+end
+
+-- Native block divs (Pandoc recognized <div class="..."> as Div)
+function Div(el)
+ if has_class(el.classes, 'only-jekyll') then
+ return {} -- drop entirely
+ elseif has_class(el.classes, 'only-pandoc') then
+ return el.content -- unwrap: keep inner blocks
+ end
+end
+
+-- Native inline spans
+function Span(el)
+ if has_class(el.classes, 'only-jekyll') then
+ return {}
+ elseif has_class(el.classes, 'only-pandoc') then
+ return el.content
+ end
+end
+
+-- Fallback for raw HTML wrappers when Pandoc didnโt turn them into Divs.
+function Pandoc(doc)
+ local out = List()
+ local mode = nil -- nil | 'drop' | 'keep'
+
+ local function is_open_of(txt, klass)
+ -- match <div ... class="... klass ...">
+ return txt:match('<div[^>]-class=[\'"][^\'"]-' .. klass .. '[^\'"]-[\'"]')
+ end
+
+ for _, blk in ipairs(doc.blocks) do
+ if blk.t == 'RawBlock' and blk.format:match('html') then
+ local t = blk.text
+ if is_open_of(t, 'only%-jekyll') then
+ mode = 'drop' -- drop wrapper and its inner content
+ elseif is_open_of(t, 'only%-pandoc') then
+ mode = 'keep' -- drop wrapper, keep inner content
+ elseif t:match('</div>') and mode ~= nil then
+ mode = nil
+ else
+ if not mode or mode == 'keep' then out:insert(blk) end
+ end
+ else
+ if not mode then
+ out:insert(blk)
+ elseif mode == 'keep' then
+ out:insert(blk)
+ end
+ end
+ end
+
+ return pandoc.Pandoc(out, doc.meta)
+end
Review Comment:
I dropped the PANDOC:START/END stuff. This class/div stuff is used instead.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
Review Comment:
fixed
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
Review Comment:
Section removed anyway.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
+```
+
+keeping them separate from the Jekyll site itself.
+
+---
+
+## ๐งฎ Example Commands
+
+From inside the `_pandoc/` directory:
+
+### Build all PDFs
+```bash
+make
+```
+
+### Clean all generated PDFs
+```bash
+make clean
+```
+
+### List all Markdown files with `pdf: true`
+```bash
+make list
+```
+
+### Force rebuild of one PDF
+```bash
+make ../about.pdf
Review Comment:
fixed
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
+```
+
+keeping them separate from the Jekyll site itself.
+
+---
+
+## ๐งฎ Example Commands
+
+From inside the `_pandoc/` directory:
+
+### Build all PDFs
+```bash
+make
+```
+
+### Clean all generated PDFs
+```bash
+make clean
+```
+
+### List all Markdown files with `pdf: true`
+```bash
+make list
+```
+
+### Force rebuild of one PDF
+```bash
+make ../about.pdf
+```
+
+---
+
+## ๐งฐ How the AWK Script Works
+
+`unwrap-pandoc.awk` removes the HTML comment wrappers used to hide LaTeX from Jekyll.
+
+Input example:
+
+````markdown
+<!-- PANDOC:START -->
+<!--
+\LaTeX code
+-->
+<!-- PANDOC:END -->
+````
+
+Output to Pandoc:
+
+```markdown
+\LaTeX code
+```
+
+That means Pandoc receives clean, valid LaTeX syntax while Jekyll never sees it.
+
+---
+
+## โ๏ธ Customizing Pandoc
+
+Pandoc is run with `--defaults=basic.yaml` which specifies the `template_basic.tex` is used.
+The template can be modified to change the PDF output.
+
+---
+
+## ๐งฑ Recommended Workflow
+
+1. Write Markdown pages normally for your Jekyll site.
+2. When you also want a PDF version, add `pdf: true` to front matter.
+3. If needed, wrap LaTeX-specific content in `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` blocks.
+4. From `_pandoc/`, run:
+ ```bash
+ make
+ ```
+5. Find the generated PDFs in `pdf/`.
+
+---
+
+## ๐ช Why This Setup Works
+
+| Concern | Solution |
+|----------|-----------|
+| Jekyll shouldnโt see LaTeX | Hidden in HTML comments |
+| Pandoc must see LaTeX | AWK removes wrappers |
+| Need automatic PDF generation | Makefile scans for `pdf: true` |
+| Keep tools separate | Everything lives in `_pandoc/` |
+
+---
+
+## ๐งพ Example Output
+
+```
+pdf/
+โโโ about.pdf
+โโโ _posts/
+ โโโ 2025-01-01-example.pdf
+```
+
+---
+
+**Maintainer Notes**
+
+- `_pandoc/Makefile` assumes itโs run from `_pandoc/`, with site root as `..`
+- Pandoc and AWK must be available on your `PATH`
+
+---
+
+## Pandoc Tools Installation
+
+These tools run on Linux.
+
+On Ubuntu you have to install these things:
+
+ sudo apt install pandoc texlive-latex-base texlive-latex-recommended \
+ texlive-fonts-recommended texlive-xetex texlive-latex-extra
+
+I have found one must update pandoc to a more up to date version.
+This is currently dependent on pandoc 3.7.0.2 which can be downloaded from
+https://github.com/jgm/pandoc/releases/tag/3.7.0.2 .
Review Comment:
I am retesting with pandoc 3.1.3.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
+```
+
+keeping them separate from the Jekyll site itself.
+
+---
+
+## ๐งฎ Example Commands
+
+From inside the `_pandoc/` directory:
+
+### Build all PDFs
+```bash
+make
+```
+
+### Clean all generated PDFs
+```bash
+make clean
+```
+
+### List all Markdown files with `pdf: true`
+```bash
+make list
+```
+
+### Force rebuild of one PDF
+```bash
+make ../about.pdf
+```
+
+---
+
+## ๐งฐ How the AWK Script Works
+
+`unwrap-pandoc.awk` removes the HTML comment wrappers used to hide LaTeX from Jekyll.
+
+Input example:
+
+````markdown
+<!-- PANDOC:START -->
+<!--
+\LaTeX code
+-->
+<!-- PANDOC:END -->
+````
+
+Output to Pandoc:
+
+```markdown
+\LaTeX code
+```
+
+That means Pandoc receives clean, valid LaTeX syntax while Jekyll never sees it.
+
+---
+
+## โ๏ธ Customizing Pandoc
+
+Pandoc is run with `--defaults=basic.yaml` which specifies the `template_basic.tex` is used.
+The template can be modified to change the PDF output.
+
+---
+
+## ๐งฑ Recommended Workflow
+
+1. Write Markdown pages normally for your Jekyll site.
+2. When you also want a PDF version, add `pdf: true` to front matter.
+3. If needed, wrap LaTeX-specific content in `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` blocks.
+4. From `_pandoc/`, run:
+ ```bash
+ make
+ ```
+5. Find the generated PDFs in `pdf/`.
+
+---
+
+## ๐ช Why This Setup Works
+
+| Concern | Solution |
+|----------|-----------|
+| Jekyll shouldnโt see LaTeX | Hidden in HTML comments |
+| Pandoc must see LaTeX | AWK removes wrappers |
+| Need automatic PDF generation | Makefile scans for `pdf: true` |
+| Keep tools separate | Everything lives in `_pandoc/` |
+
+---
Review Comment:
removed
##########
site/_pandoc/Makefile:
##########
@@ -0,0 +1,63 @@
+# ==========================================================
+# Pandoc PDF generator for Jekyll site
+# Scans Markdown files with "pdf: true" in YAML front matter
+# and produces PDFs in the site's ./pdf/ directory
+# ==========================================================
+
+# --- Configuration ---
+SITE_ROOT := ..
+AWK_UNWRAP := $(SITE_ROOT)/_pandoc/unwrap-pandoc.awk
+AWK_LIST := $(SITE_ROOT)/_pandoc/list-pdf-sources.awk
+PANDOC := pandoc
+
+# Output directory for generated PDFs (at site root)
+PDF_OUTDIR := $(SITE_ROOT)/pdf
+
+DEFAULTS := $(SITE_ROOT)/_pandoc/basic.yaml
+
+# --- Candidate Markdown files (exclude build/tool/output dirs) ---
+# Use find + awk pipeline โ awk -f avoids executable bit.
+MD_CANDIDATES := $(shell find $(SITE_ROOT) \
+ -type f -name '*.md' \
+ -not -path '*/_*/*' \
+ -not -path '*/node_modules/*' \
+ -not -path '*/vendor/*' \
+ -not -path '*/pdf/*' \
+ -print0 | xargs -0 -r awk -f $(AWK_LIST))
+
+# --- Files to build ---
+PDF_SRCS := $(MD_CANDIDATES)
+PDFS := $(patsubst $(SITE_ROOT)/%.md,$(PDF_OUTDIR)/%.pdf,$(PDF_SRCS))
+
Review Comment:
printing web pages to PDF generally results in non-useful stuff.
Making one PDF is feasible by feeding all the "pdf: true" markdown files to pandoc at once in the right sequence. I will experiment with this more later if I decide to keep pursuing this.
##########
site/pdf/dfdl-extensions.pdf:
##########
Review Comment:
I've not fixed this yet, nor the issue of one-big PDF vs. separate PDFs per page.
This is an interesting experiment in jekyll + pandoc, but I think I will focus on content creation, not this sort of tooling for a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updating documentation on site - PDF creation for some files [daffodil-site]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #193:
URL: https://github.com/apache/daffodil-site/pull/193#discussion_r2465690243
##########
site/_pandoc/Makefile:
##########
@@ -0,0 +1,63 @@
+# ==========================================================
+# Pandoc PDF generator for Jekyll site
+# Scans Markdown files with "pdf: true" in YAML front matter
+# and produces PDFs in the site's ./pdf/ directory
+# ==========================================================
+
+# --- Configuration ---
+SITE_ROOT := ..
+AWK_UNWRAP := $(SITE_ROOT)/_pandoc/unwrap-pandoc.awk
+AWK_LIST := $(SITE_ROOT)/_pandoc/list-pdf-sources.awk
+PANDOC := pandoc
+
+# Output directory for generated PDFs (at site root)
+PDF_OUTDIR := $(SITE_ROOT)/pdf
+
+DEFAULTS := $(SITE_ROOT)/_pandoc/basic.yaml
+
+# --- Candidate Markdown files (exclude build/tool/output dirs) ---
+# Use find + awk pipeline โ awk -f avoids executable bit.
+MD_CANDIDATES := $(shell find $(SITE_ROOT) \
+ -type f -name '*.md' \
+ -not -path '*/_*/*' \
+ -not -path '*/node_modules/*' \
+ -not -path '*/vendor/*' \
+ -not -path '*/pdf/*' \
+ -print0 | xargs -0 -r awk -f $(AWK_LIST))
Review Comment:
We don't have a vendor directory, and I think the pdf directory shouldn't contain and .md files? And I think node_modules is in the repo root so shouldn't be in the site root and will be ignored. Also, `_*` directories should never have pdf:true in them. All this to say, feels like we could simplify this quite a bit and just use grep, e.g.:
```bash
MD_CANDIDATES=`$(grep -Rl '^pdf: true$' $(SITE_ROOT) --include '*.md')
```
It's not as relaxed about whitespace like the awk/sed script, and doesn't even require pdf:true to be in the header, but I think that's probably fine, I doubt we'll ever have a single line with jsut "pdf: true" in it anywhere. And is much easier to maintain than the awk script.
##########
site/dfdl-extensions.md:
##########
@@ -21,38 +22,60 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
+<!--
+The :target="_blank" syntax below makes this open in a new tab
+and work in the PDF and jekyll web page.
+But displays as literal text in the IDE markdown previewer.
+-->
+<div class="only-jekyll" markdown="1">
+_This page is available as a [downloadable PDF](../pdf/dfdl-extensions.pdf){:target="_blank"}._
Review Comment:
Wondering if we can move this logic somewhere else so we don't have to duplicate it every time we mark a page as pdf: true. Maybe this is rare enough that's it's not necessary? But for example, you could put somethign like this in _navigation.html.
```
{% if page.pdf == "true" %}
<div><i>This page is available as a <a href="../pdf/{{ page.title }}.pdf" target="_blank">downloadable PDF</a></i></div>
{% endif %}
```
##########
site/_pandoc/list-pdf-sources.awk:
##########
@@ -0,0 +1,59 @@
+#!/usr/bin/awk -f
+# Prints FILENAME iff the file has YAML front matter with "pdf: true".
+# - Must be called with filenames (works via find/xargs).
+# - Ignores matches outside the front matter.
+# - Front matter is the lines between the first '---' and the next '---'.
+
+# We process each file independently.
+# Use a per-file BEGINFILE block if available (GNU awk). Otherwise reset on first record.
+BEGIN {
+ have_beginfile = 0
+}
+BEGINFILE {
+ have_beginfile = 1
+ in_front = 0
+ seen_start = 0
+ want_pdf = 0
+}
+
+# For non-GNU awk compatibility:
+# Reset on the first line of each file if BEGINFILE isn't supported.
+FNR == 1 && !have_beginfile {
+ in_front = 0
+ seen_start = 0
+ want_pdf = 0
+}
+
+{
+ # Detect start of front matter
+ if (!seen_start) {
+ if ($0 ~ /^[[:space:]]*---[[:space:]]*$/) {
+ seen_start = 1
+ in_front = 1
+ next
+ } else {
+ # No front matter: skip file
+ nextfile
+ }
+ }
+
+ # If in front matter, look for end and for pdf:true
+ if (in_front) {
+ if ($0 ~ /^[[:space:]]*---[[:space:]]*$/) {
+ # End of front matter
+ in_front = 0
+ if (want_pdf) {
+ print FILENAME
+ }
+ nextfile
+ }
+ # Match "pdf: true" allowing spaces; ensure it's a key at line start
+ if ($0 ~ /^[[:space:]]*pdf:[[:space:]]*true([[:space:]]|$)/) {
+ want_pdf = 1
+ }
+ next
+ }
+
+ # If we got here, weโve passed front matter without finding pdf:true
+ nextfile
+}
Review Comment:
Thoughts on a simple sed/bash script?
```bash
#!/bin/bash
for FILE in "$@"; do
sed -n '/^---$/,/^---$/p' "$FILE" |
grep -Eq '^[[:space:]]*pdf:[[:space:]]*true[[:space:]]*$' && echo "$FILE"
done
```
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
Review Comment:
Can we remove these emojis? I don't think they add anything, if anythign they make things more confusing.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
Review Comment:
These don't exist, can we remove these?
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
Review Comment:
We don't have config.yml or _post or _pages. Suggest we just remove this as it doesn't provide anything useful.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
+```
+
+keeping them separate from the Jekyll site itself.
+
+---
+
+## ๐งฎ Example Commands
+
+From inside the `_pandoc/` directory:
+
+### Build all PDFs
+```bash
+make
+```
+
+### Clean all generated PDFs
+```bash
+make clean
+```
+
+### List all Markdown files with `pdf: true`
+```bash
+make list
+```
+
+### Force rebuild of one PDF
+```bash
+make ../about.pdf
+```
+
+---
+
+## ๐งฐ How the AWK Script Works
+
+`unwrap-pandoc.awk` removes the HTML comment wrappers used to hide LaTeX from Jekyll.
+
+Input example:
+
+````markdown
+<!-- PANDOC:START -->
+<!--
+\LaTeX code
+-->
+<!-- PANDOC:END -->
+````
+
+Output to Pandoc:
+
+```markdown
+\LaTeX code
+```
+
+That means Pandoc receives clean, valid LaTeX syntax while Jekyll never sees it.
+
+---
Review Comment:
I think the aboe section kindof already mentions pandoc:start/end.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
+```
+
+keeping them separate from the Jekyll site itself.
+
+---
+
+## ๐งฎ Example Commands
+
+From inside the `_pandoc/` directory:
+
+### Build all PDFs
+```bash
+make
+```
+
+### Clean all generated PDFs
+```bash
+make clean
+```
+
+### List all Markdown files with `pdf: true`
+```bash
+make list
+```
+
+### Force rebuild of one PDF
+```bash
+make ../about.pdf
+```
+
+---
+
+## ๐งฐ How the AWK Script Works
+
+`unwrap-pandoc.awk` removes the HTML comment wrappers used to hide LaTeX from Jekyll.
+
+Input example:
+
+````markdown
+<!-- PANDOC:START -->
+<!--
+\LaTeX code
+-->
+<!-- PANDOC:END -->
+````
+
+Output to Pandoc:
+
+```markdown
+\LaTeX code
+```
+
+That means Pandoc receives clean, valid LaTeX syntax while Jekyll never sees it.
+
+---
+
+## โ๏ธ Customizing Pandoc
+
+Pandoc is run with `--defaults=basic.yaml` which specifies the `template_basic.tex` is used.
+The template can be modified to change the PDF output.
+
+---
+
+## ๐งฑ Recommended Workflow
+
+1. Write Markdown pages normally for your Jekyll site.
+2. When you also want a PDF version, add `pdf: true` to front matter.
+3. If needed, wrap LaTeX-specific content in `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` blocks.
+4. From `_pandoc/`, run:
+ ```bash
+ make
+ ```
+5. Find the generated PDFs in `pdf/`.
+
+---
+
+## ๐ช Why This Setup Works
+
+| Concern | Solution |
+|----------|-----------|
+| Jekyll shouldnโt see LaTeX | Hidden in HTML comments |
+| Pandoc must see LaTeX | AWK removes wrappers |
+| Need automatic PDF generation | Makefile scans for `pdf: true` |
+| Keep tools separate | Everything lives in `_pandoc/` |
+
+---
Review Comment:
Suggest we remove this section.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
+```
+
+keeping them separate from the Jekyll site itself.
+
+---
+
+## ๐งฎ Example Commands
+
+From inside the `_pandoc/` directory:
+
+### Build all PDFs
+```bash
+make
+```
+
+### Clean all generated PDFs
+```bash
+make clean
+```
+
+### List all Markdown files with `pdf: true`
+```bash
+make list
+```
+
+### Force rebuild of one PDF
+```bash
+make ../about.pdf
+```
+
+---
+
+## ๐งฐ How the AWK Script Works
+
+`unwrap-pandoc.awk` removes the HTML comment wrappers used to hide LaTeX from Jekyll.
+
+Input example:
+
+````markdown
+<!-- PANDOC:START -->
+<!--
+\LaTeX code
+-->
+<!-- PANDOC:END -->
+````
+
+Output to Pandoc:
+
+```markdown
+\LaTeX code
+```
+
+That means Pandoc receives clean, valid LaTeX syntax while Jekyll never sees it.
+
+---
+
+## โ๏ธ Customizing Pandoc
+
+Pandoc is run with `--defaults=basic.yaml` which specifies the `template_basic.tex` is used.
+The template can be modified to change the PDF output.
+
+---
+
+## ๐งฑ Recommended Workflow
+
+1. Write Markdown pages normally for your Jekyll site.
+2. When you also want a PDF version, add `pdf: true` to front matter.
+3. If needed, wrap LaTeX-specific content in `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` blocks.
+4. From `_pandoc/`, run:
+ ```bash
+ make
+ ```
+5. Find the generated PDFs in `pdf/`.
+
+---
+
+## ๐ช Why This Setup Works
+
+| Concern | Solution |
+|----------|-----------|
+| Jekyll shouldnโt see LaTeX | Hidden in HTML comments |
+| Pandoc must see LaTeX | AWK removes wrappers |
+| Need automatic PDF generation | Makefile scans for `pdf: true` |
+| Keep tools separate | Everything lives in `_pandoc/` |
+
+---
+
+## ๐งพ Example Output
+
+```
+pdf/
+โโโ about.pdf
+โโโ _posts/
+ โโโ 2025-01-01-example.pdf
+```
+
+---
+
+**Maintainer Notes**
+
+- `_pandoc/Makefile` assumes itโs run from `_pandoc/`, with site root as `..`
+- Pandoc and AWK must be available on your `PATH`
+
+---
+
+## Pandoc Tools Installation
+
+These tools run on Linux.
+
+On Ubuntu you have to install these things:
+
+ sudo apt install pandoc texlive-latex-base texlive-latex-recommended \
+ texlive-fonts-recommended texlive-xetex texlive-latex-extra
+
+I have found one must update pandoc to a more up to date version.
+This is currently dependent on pandoc 3.7.0.2 which can be downloaded from
+https://github.com/jgm/pandoc/releases/tag/3.7.0.2 .
Review Comment:
Does this really depend on 3.7? I don't think we should rely on devs needing to install newer versions from source. Fedora only ships with 3.1, suggest we avoid modern features to make these easier to test.
##########
site/dfdl-extensions.md:
##########
@@ -21,38 +22,60 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
+<!--
+The :target="_blank" syntax below makes this open in a new tab
+and work in the PDF and jekyll web page.
+But displays as literal text in the IDE markdown previewer.
+-->
+<div class="only-jekyll" markdown="1">
+_This page is available as a [downloadable PDF](../pdf/dfdl-extensions.pdf){:target="_blank"}._
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+</div>
+
+<div class="only-pandoc" markdown="1">
+# Introduction
+</div>
Review Comment:
Shoudl we just always have the Intruduction header? The less differences between PDF and website the better.
##########
site/dfdl-extensions.md:
##########
@@ -21,38 +22,60 @@ See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
+<!--
+The :target="_blank" syntax below makes this open in a new tab
+and work in the PDF and jekyll web page.
+But displays as literal text in the IDE markdown previewer.
+-->
+<div class="only-jekyll" markdown="1">
+_This page is available as a [downloadable PDF](../pdf/dfdl-extensions.pdf){:target="_blank"}._
+
+### Table of Contents
+{:.no_toc}
+
+1. use ordered table of contents
+{:toc}
+</div>
+
+<div class="only-pandoc" markdown="1">
+# Introduction
+</div>
Daffodil provides extensions to the DFDL specification.
-These properties are in the namespace defined by the URI
+These functions and properties are in the namespace defined by the URI
``http://www.ogf.org/dfdl/dfdl-1.0/extensions`` which is normally bound to the ``dfdlx`` prefix
like so:
``` xml
-<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
- xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+<schema xmlns="http://www.w3.org/2001/XMLSchema"
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
>
```
-The following symbols defined in this namespace are described below.
+The DFDL language extensions described below have Long Term Support (LTS) in Daffodil
+going forward, and are proposed for inclusion in a future revision of the DFDL
+standard.
+DFDL schema authors can depend on the features and behaviors defined here without fear
+that these extensions will be withdrawn in the future.
-### Expression Functions
+# Expression Functions
Review Comment:
I think there wareason we used the `###`. That said, we should probably figure out why and change things. It is pretty annoying that our .md pages cant use normal headings. I'll look into this and see if I can figure out the reasoning and if we can change it.
##########
site/dfdl-extensions.md:
##########
@@ -87,46 +110,112 @@ found after fields `a` and `b`:
<xs:element name="tag" type="xs:int" dfdl:length="8" />
```
-Bitwise Functions
+## Bitwise Functions: `bitAnd`, `bitOr`, `bitXor`, `bitNot`, `leftShift`, `rightShift`
+
+These functions are defined on types `long`, `int`, `short`, `byte`, `unsignedLong`,
+`unsignedInt`, `unsignedShort`, and `unsignedByte`
+
+### `dfdlx:bitAnd(arg1, arg2)`
+
+This computes the bitwise AND of two integers.
+
+- Both arguments must be signed, or both must be unsigned.
+- If the two arguments are not the same type the smaller one is converted into the type of the
+larger one.
+- If the smaller argument is signed, this conversion does sign-extension.
+- The result type is the that of the largest argument.
+
+### `dfdlx:bitOr(arg1, arg2)`
+
+This computes the bitwise OR of two integers.
+
+- Both arguments must be signed, or both must be unsigned.
+- If the two arguments are not the same type the smaller one is converted into the type of the
+larger one.
+- If the smaller argument is signed, this conversion does sign-extension.
+- The result type is the that of the largest argument.
+
+### `dfdlx:bitXor(arg1, arg2)`
+
+This computes the bitwise Exclusive OR of two integers.
+
+- Both arguments must be signed, or both must be unsigned.
+- If the two arguments are not the same type the smaller one is converted into the type of the
+larger one.
+- If the smaller argument is signed, this conversion does sign-extension.
+- The result type is the that of the largest argument.
+
+### `dfdlx:bitNot(arg)`
+
+This computes the bitwise NOT of an integer. Every bit is inverted. The result type is the same
+as the argument type.
+
+### `dfdlx:leftShift(value, shiftCount)`
+
+This is the _logical_ shift left, meaning that bits are shifted from less-significant positions
+to more-significant positions.
+
+- The left-most bits shifted out are discarded.
+- Zeros are shifted in for the right-most bits.
+- The result type is the same as the `value` argument type.
+- It is a processing error if the `shiftCount` argument is < 0.
+- It is a processing error if the `shiftCount` argument is greater than the number of
+ bits in the type of the value argument.
+
+### `dfdlx:rightShift(value, shiftCount)`
+
+This is the _arithmetic_ shift right, meaning bits move from most-significant to
+less-significant positions.
+If _logical_ (zero-filling) shift right is needed, you must use unsigned types.
+
+- The `value` argument is shifted by the `shiftCount`.
+- The right-most bits shifted out are discarded.
+- If the `value` is signed, then the sign bit is shifted in for the left-most bits.
+- If the `value` is unsigned, then zeros are shifted in for the left-most bits.
+- The result type is the same as the `value` argument type.
+- It is a processing error if the `shiftCount` argument is < 0.
+- It is a processing error if the `shiftCount` argument is greater than the number of
+ bits in the type of the value argument.
+
+## `dfdlx:doubleFromRawLong(longArg): double` and `dfdlx:doubleToRawLong(doubleArg): long`
- : TBD, but the complete list (all ``dfdlx``) is `BitAnd`, `BitNot`, `BitOr`, `BitXor`, `LeftShift`,
- `RightShift`
+IEEE binary float and double values that are not NaN will parse to base 10 text and unparse back
+to the same exact IEEE binary bits.
+However, the same cannot be said for NaN (not a number) values, of which there are many bit
+patterns.
+To preserve float and double NaN values bit for bit you can use these functions to compute
+`xs:long` values that enable the DFDL Infoset to preserve the bits of a float or double value
+even if it is a NaN.
-``dfdlx:doubleFromRawLong`` and ``dfdlx:doubleToRawLong``
- : Converting binary floating point numbers to/from base 10 text can result in lost information.
-The base 10 representation, converted back to binary representation, may not be bit-for-bit
- identical. These functions can be used to carry 8-byte double precision IEEE floating point
- numbers as type `xs:long` so that no information is lost. The DFDL schema can still obtain
- and operate on the floating point value by converting these `xs:long` values into type
- `xs:double`, and back if necessary for unparsing a new value.
-### Properties
+# Properties
-``dfdlx:parseUnparsePolicy``
+## `dfdlx:parseUnparsePolicy`
- : A property applied to simple and complex elements, which specifies whether the element supports only parsing, only unparsing, or both parsing and unparse. Valid values for this property are ``parse``, ``unparse``, or ``both``. This allows one to leave off properties that are required for only parse or only unparse, such as ``dfdl:outputValueCalc`` or ``dfdl:outputNewLine``, so that one may have a valid schema if only a subset of functionality is needed.
+A property applied to simple and complex elements, which specifies whether the element supports only parsing, only unparsing, or both parsing and unparse. Valid values for this property are ``parse``, ``unparse``, or ``both``. This allows one to leave off properties that are required for only parse or only unparse, such as ``dfdl:outputValueCalc`` or ``dfdl:outputNewLine``, so that one may have a valid schema if only a subset of functionality is needed.
- All elements must have a compatible parseUnparsePolicy with the compilation parseUnparsePolicy (which is defined by the root element daf:parseUnparsePolicy and/or the Daffodil parseUnparsePolicy tunable) or it is a Schema Definition Error. An element is defined to have a compatible parseUnparsePolicy if it has the same value as the compilation parseUnparsePolicy or if it has the value ``both``.
+All elements must have a compatible parseUnparsePolicy with the compilation parseUnparsePolicy (which is defined by the root element daf:parseUnparsePolicy and/or the Daffodil parseUnparsePolicy tunable) or it is a Schema Definition Error. An element is defined to have a compatible parseUnparsePolicy if it has the same value as the compilation parseUnparsePolicy or if it has the value ``both``.
- For compatibility, if this property is not defined, it is assumed to be ``both``.
+For compatibility, if this property is not defined, it is assumed to be ``both``.
-``dfdlx:layer``
+## `dfdlx:layer`
- : [Layers](/layers) provide algorithmic capabilities for decoding/encoding data or computing
+_Layers_ provide algorithmic capabilities for decoding/encoding data or computing
checksums. Some are built-in to Daffodil. New layers can be created in Java/Scala and
plugged-in to Daffodil dynamically.
+There is [separate Layer documentation](/layers).
-``dfdlx:direction``
+## `dfdlx:direction`
- : TBD
+This property has
Review Comment:
Should probably stay TBD until documented.
##########
site/_pandoc/only.lua:
##########
@@ -0,0 +1,60 @@
+-- only.lua: drop .only-jekyll, keep contents of .only-pandoc
+-- Handles both native Div/Span nodes and raw HTML <div> wrappers.
+
+local List = require 'pandoc.List'
+
+local function has_class(classes, cls)
+ return classes and List.includes(classes, cls)
+end
+
+-- Native block divs (Pandoc recognized <div class="..."> as Div)
+function Div(el)
+ if has_class(el.classes, 'only-jekyll') then
+ return {} -- drop entirely
+ elseif has_class(el.classes, 'only-pandoc') then
+ return el.content -- unwrap: keep inner blocks
+ end
+end
+
+-- Native inline spans
+function Span(el)
+ if has_class(el.classes, 'only-jekyll') then
+ return {}
+ elseif has_class(el.classes, 'only-pandoc') then
+ return el.content
+ end
+end
+
+-- Fallback for raw HTML wrappers when Pandoc didnโt turn them into Divs.
+function Pandoc(doc)
+ local out = List()
+ local mode = nil -- nil | 'drop' | 'keep'
+
+ local function is_open_of(txt, klass)
+ -- match <div ... class="... klass ...">
+ return txt:match('<div[^>]-class=[\'"][^\'"]-' .. klass .. '[^\'"]-[\'"]')
+ end
+
+ for _, blk in ipairs(doc.blocks) do
+ if blk.t == 'RawBlock' and blk.format:match('html') then
+ local t = blk.text
+ if is_open_of(t, 'only%-jekyll') then
+ mode = 'drop' -- drop wrapper and its inner content
+ elseif is_open_of(t, 'only%-pandoc') then
+ mode = 'keep' -- drop wrapper, keep inner content
+ elseif t:match('</div>') and mode ~= nil then
+ mode = nil
+ else
+ if not mode or mode == 'keep' then out:insert(blk) end
+ end
+ else
+ if not mode then
+ out:insert(blk)
+ elseif mode == 'keep' then
+ out:insert(blk)
+ end
+ end
+ end
+
+ return pandoc.Pandoc(out, doc.meta)
+end
Review Comment:
What is the difference between these .only-pandoc and the PANDOC:START/END things? Seems like they are two different ways to do the same thign?
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
+```
+
+keeping them separate from the Jekyll site itself.
+
+---
+
+## ๐งฎ Example Commands
+
+From inside the `_pandoc/` directory:
+
+### Build all PDFs
+```bash
+make
+```
+
+### Clean all generated PDFs
+```bash
+make clean
+```
+
+### List all Markdown files with `pdf: true`
+```bash
+make list
+```
+
+### Force rebuild of one PDF
+```bash
+make ../about.pdf
Review Comment:
Is this right, shouldn't this be make ../pdf/about.pdf?
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
+
+Example:
+
+````markdown
+Regular Markdown content here.
+
+<!-- PANDOC:START -->
+<!--
+```{=latex}
+\begin{tabular}{ll}
+A & B \\
+C & D \\
+\end{tabular}
+```
+-->
+<!-- PANDOC:END -->
+
+More Markdown content.
+````
+
+When viewed on the Jekyll site:
+- This section is hidden (HTML comments are ignored).
+
+When built via Pandoc:
+- The `unwrap-pandoc.awk` script strips the comment wrappers,
+- The inner LaTeX becomes active, producing a correct PDF table.
+
+---
+
+### 3. The Makefile
+
+The `_pandoc/Makefile` automates the whole process.
+
+It:
+
+1. Recursively scans the site for Markdown files with `pdf: true`.
+2. Runs `unwrap-pandoc.awk` to clean up `<!-- PANDOC:START -->` / `<!-- PANDOC:END -->` wrappers.
+3. Invokes Pandoc with the configured LaTeX template to produce a PDF.
+
+The resulting PDFs go into:
+
+```
+_pandoc/output/
Review Comment:
I thought they go int he ../pdf directory?
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
Review Comment:
Do we need this? I don't think we actually use latex anywhere. I also would want to avoid things where are pdf files are different than our markdown. Seems like pandoc should be able to handle converting our normal markdown files to pdf without a problem. It would be nice if we could remove all the preprocessor stuff if we don't really use it.
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
Review Comment:
We don't have _posts or _pages, suggest we just say .md files.
##########
site/pdf/dfdl-extensions.pdf:
##########
Review Comment:
We should not commit changes to these PDF files. Instead, we should modify the build/publish CI tool to rebuild the PDF files and commit them along with site changes.
##########
site/_pandoc/Makefile:
##########
@@ -0,0 +1,63 @@
+# ==========================================================
+# Pandoc PDF generator for Jekyll site
+# Scans Markdown files with "pdf: true" in YAML front matter
+# and produces PDFs in the site's ./pdf/ directory
+# ==========================================================
+
+# --- Configuration ---
+SITE_ROOT := ..
+AWK_UNWRAP := $(SITE_ROOT)/_pandoc/unwrap-pandoc.awk
+AWK_LIST := $(SITE_ROOT)/_pandoc/list-pdf-sources.awk
+PANDOC := pandoc
+
+# Output directory for generated PDFs (at site root)
+PDF_OUTDIR := $(SITE_ROOT)/pdf
+
+DEFAULTS := $(SITE_ROOT)/_pandoc/basic.yaml
+
+# --- Candidate Markdown files (exclude build/tool/output dirs) ---
+# Use find + awk pipeline โ awk -f avoids executable bit.
+MD_CANDIDATES := $(shell find $(SITE_ROOT) \
+ -type f -name '*.md' \
+ -not -path '*/_*/*' \
+ -not -path '*/node_modules/*' \
+ -not -path '*/vendor/*' \
+ -not -path '*/pdf/*' \
+ -print0 | xargs -0 -r awk -f $(AWK_LIST))
+
+# --- Files to build ---
+PDF_SRCS := $(MD_CANDIDATES)
+PDFS := $(patsubst $(SITE_ROOT)/%.md,$(PDF_OUTDIR)/%.pdf,$(PDF_SRCS))
+
Review Comment:
I wonder how useful it is to have a single PDF per page. For examle, if someone wants to create a PDF of a page, they can just print it to a PDF.
Feels like a more useful documentation would be to combine chosen pages into a single large PDF. That way users could download that and have a single offlien source for all Daffodil releated things. That might simplify much of the logic too. For example, we just have one navigation link do download offline documentation, rather than having a bunch of pages with indivdual links.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updating documentation on site - PDF creation for some files [daffodil-site]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #193:
URL: https://github.com/apache/daffodil-site/pull/193#discussion_r2470115814
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
Review Comment:
Yeah, feels like maybe jekyll isn't the best tool for this kind of documentation--it's really just too focused on website generation.
Maybe what we need is a tool that is primarily about creating documentation rather than building websites? Something that already supports complex tables, diagrams, etc. and can generate PDFs and/or HTML. Then we can generate the PDF for offline use and HTML for online use. And it's fine if the HTML styling was different from our normal website--we already have Javadocs that use a different style, so it would just be similar to that. And then maybe all of the stuff under the current "Docs" navigation gets migrated to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updating documentation on site - PDF creation for some files [daffodil-site]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle closed pull request #193: Updating documentation on site - PDF creation for some files
URL: https://github.com/apache/daffodil-site/pull/193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Updating documentation on site - PDF creation for some files [daffodil-site]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #193:
URL: https://github.com/apache/daffodil-site/pull/193#discussion_r2469107921
##########
site/_pandoc/README.md:
##########
@@ -0,0 +1,223 @@
+---
+layout: page
+title: Pandoc + Jekyll Integration
+pdf: false
+---
+# ๐งญ Pandoc + Jekyll Integration
+
+This directory contains tools for generating **PDF versions** of selected Jekyll pages while keeping the same Markdown files usable by Jekyll for the website.
+
+The goal is to have **one Markdown source** that:
+- renders cleanly in the Jekyll site (for HTML),
+- and can also be converted into a polished PDF using **Pandoc + LaTeX**.
+
+---
+
+## ๐๏ธ Directory Layout
+
+```
+_pandoc/
+โ
+โโโ README.md โ this file
+โโโ Makefile โ builds all PDFs
+โโโ unwrap-pandoc.awk โ preprocessor that removes comment wrappers
+โโโ template.latex โ (optional) custom LaTeX template
+โโโ header.tex โ (optional) extra LaTeX header content
+โโโ ../pdf/ โ generated PDFs appear here
+```
+
+At the root of the Jekyll site:
+
+```
+_config.yml
+_posts/
+pages/
+assets/
+_pandoc/
+pdf/
+```
+
+---
+
+## ๐งฉ How It Works
+
+### 1. Mark pages that should have PDFs
+
+Any Markdown file (in `_posts`, `pages/`, or elsewhere) can be tagged with:
+
+```yaml
+---
+title: Example Page
+layout: page
+pdf: true
+---
+```
+
+The Makefile will scan the entire Jekyll project and automatically detect these files.
+
+---
+
+### 2. Use HTML comment wrappers for Pandoc-only content
+
+Pandoc sometimes needs LaTeX code for things like custom tables, math, or page layout.
+We hide that LaTeX from Jekyll using **HTML comments**, which Jekyll ignores but our AWK preprocessor removes before running Pandoc.
Review Comment:
This was needed in the relnotes for some DFDL Schema project that I cloned this approach from. Even just doing the table of revisions at the start of the document required more than just markdown tables because there is a notes column that is a paragraph.
This is needed if you want to create almost any sort of rich tables that work like the ones in the DFDL specification, where a cell of the table contains rich structure like bulleted or numbered lists, etc.
Markdown is really quite restrictive about tables. Even putting a basic paragraph into a table cell is problematic, I think because a complex table cell isn't that readable when presented as the ".md" text, nor easy to create, so this kind of rich nesting structure is against the markdown philosophy. Jekyll lets you use HTML for tables, but pandoc does not.
This is actually the primary issue that is making me question this whole approach.
If I can't make a stylistically similar PDF to the DFDL spec, then I'm not sure what the point is. We could easily package the whole daffodil site so that offline users can just run a local copy of the whole thing... in fact, just git clone of daffodil-site and issuing the podman command will do this. (though there may be a simpler way using wget for example).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org