You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "bitflicker64 (via GitHub)" <gi...@apache.org> on 2026/02/09 12:43:45 UTC
[PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
bitflicker64 opened a new pull request, #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452
<!--
Thank you very much for contributing to Apache HugeGraph, we are happy that you want to help us improve it!
Some tips for you:
1. If this is your first time to submit PR, please read the
[contributing guidelines](https://github.com/apache/incubator-hugegraph-doc/blob/master/contribution.md)
2. If a PR fix/close an issue, type the message "close xxx" below (Remember to update both EN & CN doc)
3. Build the website locally after you finish the PR, and check if the changes are correct, THX~
-->
## Purpose of the PR
- close #451 <!-- or "fix #xxx", "link #xxx" -->
<!-- Better to paste the screenshot diff here, "xxx" is the ID-link of related issue, e.g: #1024 -->
### What this PR does
This PR hardens the documentation link validation used in CI.
Previously, CI could pass even when documentation contained broken internal links due to limited link detection and path resolution. This change expands validation coverage and ensures CI fails when unresolved internal links are present.
### Key changes
- Validate all inline Markdown links with accurate file and line numbers
- Correctly resolve internal paths, including:
- `/docs/* → content/en/docs/*`
- `/cn/docs/* → content/cn/docs/*`
- absolute root paths
- relative links
- Exclude fenced code blocks from validation without breaking line numbers
- Skip external links and Hugo shortcodes intentionally
- Validate non-Markdown assets separately (images, configs, etc.)
- Fail CI deterministically on unresolved internal links
### Notes
This change surfaces existing broken documentation links that CI was previously not detecting.
Those issues can be addressed incrementally in follow-up PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on PR #452:
URL: https://github.com/apache/hugegraph-doc/pull/452#issuecomment-3958279743
> > @bitflicker64 Hi, still found some 404 links & broken image link:
> >
> > * [hugegraph.apache.org/docs/introduction/README](https://hugegraph.apache.org/docs/introduction/README) (both in EN & CN)
> > * [hugegraph.apache.org/docs/quickstart/hugegraph-ai/quick_start](https://hugegraph.apache.org/docs/quickstart/hugegraph-ai/quick_start/)
>
> Makes sense those are absolute [hugegraph.apache.org](https://hugegraph.apache.org/)... links, and the validator currently skips all http/https targets, so it won’t catch site 404s. We can either update the docs to use /docs/... (so they’re checked), or add an optional allowlist-based external check for hugegraph.apache.org. Would you prefer I add an optional allowlist , or keep CI limited to internal links and handle these in follow up fixes?
BTW, already fixed the first issue in https://github.com/apache/hugegraph-doc/pull/454/changes/b1fccc8faf7e424e1e16d059295d67e84ec726f1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#issuecomment-3871530815
This PR tightens the link validation so CI fails when unresolved internal links exist. Earlier some broken links were not being detected because the validation coverage was limited which allowed CI to pass.
With the updated validator those existing issues are now surfaced. This PR focuses on fixing the validation behaviour itself and the underlying documentation links can be cleaned up separately in follow up PRs if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786514323
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
+
+ case "$clean_lower" in
+ *.png|*.jpg|*.jpeg|*.svg|*.gif|*.xml|*.yaml|*.yml|*.json|*.css|*.js|*.pdf|*.zip|*.tar.gz)
+ if [[ -f "$target_path" ]]; then
+ return 0
+ else
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+ return
+ fi
+ ;;
+ esac
+
+ if [[ -f "$target_path" || -f "$target_path.md" || -f "$target_path/_index.md" || -f "$target_path/README.md" ]]; then
+ return 0
+ fi
+
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+}
+
+echo "Starting link validation..."
+
+while read -r FILE; do
+ declare -A CODE_LINES
+ in_fence=false
+ line_no=0
+
+ while IFS= read -r line; do
+ ((line_no++))
+
+ if [[ "$line" =~ ^[[:space:]]*(\`\`\`|~~~) ]]; then
+ if $in_fence; then
+ in_fence=false
+ else
+ in_fence=true
+ fi
+ CODE_LINES[$line_no]=1
continue
fi
- # Check for file existence variations
- FOUND=false
-
- # Check 1: As .md file
- if [[ -f "${TARGET_PATH}.md" ]]; then
- FOUND=true
- # Check 2: Exact file (if extension was included)
- elif [[ -f "$TARGET_PATH" ]]; then
- FOUND=true
- # Check 3: Directory index
- elif [[ -f "${TARGET_PATH}/_index.md" ]]; then
- FOUND=true
- # Check 4: Directory README (legacy)
- elif [[ -f "${TARGET_PATH}/README.md" ]]; then
- FOUND=true
+ if $in_fence; then
+ CODE_LINES[$line_no]=1
+ continue
Review Comment:
I’ve hardened it to ignore escaped backticks and documented the remaining limitations inline to avoid incorrect assumptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2791586905
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" || exit 1
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)" || exit 1
+
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ # Decode common URL-encoded characters explicitly
+ link="${link//%20/ }" # space
+ link="${link//%23/#}" # hash
+ link="${link//%2F/\/}" # forward slash
+
+ # Generic percent-decoding for remaining cases
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
+ else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use Python to compute realpath which resolves symlinks AND normalizes paths
+ # Python's os.path.realpath is tolerant of non-existent final targets
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback: Normalize without symlink resolution if Python3 unavailable
+ # Note: This won't resolve symlinks, only normalize .. and . components
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ log_verbose "Skipping Hugo shortcode link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ log_verbose "Skipping external link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ # Skip validation for ambiguous absolute paths (Hugo runtime URLs)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Warning: Skipping validation for ambiguous absolute path"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Reason: Hugo runtime URL (not directly mappable to filesystem)"
+ return 0
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
+
+ if [[ "$clean_lower" =~ \.(${ASSET_EXTENSIONS_REGEX})$ ]]; then
+ if [[ -f "$target_path" ]]; then
+ return 0
+ else
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+ return
+ fi
+ fi
+
+ if [[ -f "$target_path" || -f "$target_path.md" || -f "$target_path/_index.md" || -f "$target_path/README.md" ]]; then
+ return 0
+ fi
+
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+}
+
echo "Starting link validation..."
-# Find all markdown files and verify links
while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+ declare -A CODE_LINES
Review Comment:
P0: This script now depends on Bash 4+ (`declare -A` here and `${var,,}` below), but macOS default `/bin/bash` is 3.2. In that environment the script logs syntax errors and still exits 0, which can produce false "Link validation passed". Please either add an explicit version guard (`BASH_VERSINFO >= 4`, otherwise exit 1) or remove Bash-4-only syntax so failure is deterministic.
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" || exit 1
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)" || exit 1
+
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ # Decode common URL-encoded characters explicitly
+ link="${link//%20/ }" # space
+ link="${link//%23/#}" # hash
+ link="${link//%2F/\/}" # forward slash
+
+ # Generic percent-decoding for remaining cases
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
+ else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use Python to compute realpath which resolves symlinks AND normalizes paths
+ # Python's os.path.realpath is tolerant of non-existent final targets
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback: Normalize without symlink resolution if Python3 unavailable
+ # Note: This won't resolve symlinks, only normalize .. and . components
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ log_verbose "Skipping Hugo shortcode link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ log_verbose "Skipping external link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
Review Comment:
P1: This branch skips validation for every absolute path except `/docs/*` and `/cn/docs/*`. That leaves real internal links (for example `/community/maturity/`, `/clients/gremlin-console.html`, `/blog/...`) completely unchecked, so broken links can still pass CI. Please consider resolving known site-root paths to `content/` or `static/`, and failing unknown absolute paths unless they are explicitly allowlisted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2791628517
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" || exit 1
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)" || exit 1
+
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ # Decode common URL-encoded characters explicitly
+ link="${link//%20/ }" # space
+ link="${link//%23/#}" # hash
+ link="${link//%2F/\/}" # forward slash
+
+ # Generic percent-decoding for remaining cases
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
+ else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use Python to compute realpath which resolves symlinks AND normalizes paths
+ # Python's os.path.realpath is tolerant of non-existent final targets
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback: Normalize without symlink resolution if Python3 unavailable
+ # Note: This won't resolve symlinks, only normalize .. and . components
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ log_verbose "Skipping Hugo shortcode link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ log_verbose "Skipping external link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
Review Comment:
⚠️ This branch silently skips validation for absolute internal paths outside `/docs/*` and `/cn/docs/*`. That can hide broken internal links and make the check non-deterministic. Consider failing these unknown absolute paths instead of warning-and-skip.
```suggestion
elif [[ "$clean_link" == /* ]]; then
location="$file"
[[ -n "$line_no" ]] && location="$file:$line_no"
echo "Error: Unsupported absolute internal path (cannot validate deterministically)"
echo " File: $location"
echo " Link: $link"
EXIT_CODE=1
return
```
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
Review Comment:
‼️ The script currently continues after shell/runtime errors and may still print `Link validation passed!`, which risks false positives. Please fail fast so any parsing/runtime error immediately fails validation.
```suggestion
set -o errexit
set -o pipefail
```
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" || exit 1
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)" || exit 1
+
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ # Decode common URL-encoded characters explicitly
+ link="${link//%20/ }" # space
+ link="${link//%23/#}" # hash
+ link="${link//%2F/\/}" # forward slash
+
+ # Generic percent-decoding for remaining cases
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
+ else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use Python to compute realpath which resolves symlinks AND normalizes paths
+ # Python's os.path.realpath is tolerant of non-existent final targets
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback: Normalize without symlink resolution if Python3 unavailable
+ # Note: This won't resolve symlinks, only normalize .. and . components
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ log_verbose "Skipping Hugo shortcode link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ log_verbose "Skipping external link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ # Skip validation for ambiguous absolute paths (Hugo runtime URLs)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Warning: Skipping validation for ambiguous absolute path"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Reason: Hugo runtime URL (not directly mappable to filesystem)"
+ return 0
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
+
+ if [[ "$clean_lower" =~ \.(${ASSET_EXTENSIONS_REGEX})$ ]]; then
+ if [[ -f "$target_path" ]]; then
+ return 0
+ else
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+ return
+ fi
+ fi
+
+ if [[ -f "$target_path" || -f "$target_path.md" || -f "$target_path/_index.md" || -f "$target_path/README.md" ]]; then
+ return 0
+ fi
+
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+}
+
echo "Starting link validation..."
-# Find all markdown files and verify links
while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+ declare -A CODE_LINES
Review Comment:
‼️ Bash 4-only features are used in this script (for example `declare -A` here and `${clean_link,,}` below). On macOS default Bash 3.2 this fails at runtime and can still produce a false "passed" result. Please either enforce Bash 4+ explicitly or rewrite these parts with Bash 3-compatible syntax.
```suggestion
if (( BASH_VERSINFO[0] < 4 )); then
echo "Error: dist/validate-links.sh requires Bash 4+ (found ${BASH_VERSION})"
exit 2
fi
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2791588145
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
Review Comment:
P2: `tar.gz` in this regex is interpreted as `tar` + any-char + `gz`, so values like `foo.tarXgz` also match as assets. Escaping the dot (`tar\.gz`) avoids false matches and keeps extension classification precise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2782667697
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,132 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+normalize_link() {
+ local link="$1"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+
+ clean_link=$(normalize_link "$link")
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{<"* || "$clean_link" == "{{%"* || "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_link_lower="${clean_link,,}"
+
+ if [[ "$clean_link_lower" == http://* || "$clean_link_lower" == https://* || "$clean_link_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_link_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="content/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="content${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="content/en${clean_link}"
+ else
+ local file_dir
+ file_dir=$(dirname "$file")
+ target_path="${file_dir}/${clean_link}"
+
+ while [[ "$target_path" == *"/./"* ]]; do
+ target_path="${target_path//\/.\//\/}"
+ done
+
+ while [[ "$target_path" =~ ([^/]+/\.\./?) ]]; do
Review Comment:
⚠️ path normalization looks incorrect and may silently produce wrong target paths.
Current reduction:
This removes the substring but does not remove the preceding directory component the way real path canonicalization would.
Suggestion: use a more reliable canonicalization approach (e.g. ) or (if available), and ensure the result stays under .
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,132 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+normalize_link() {
+ local link="$1"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+
+ clean_link=$(normalize_link "$link")
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{<"* || "$clean_link" == "{{%"* || "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_link_lower="${clean_link,,}"
+
+ if [[ "$clean_link_lower" == http://* || "$clean_link_lower" == https://* || "$clean_link_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_link_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
Review Comment:
⚠️ should defensively handle missing/empty line numbers.
In the new extraction pipeline, comes from matches. If anything changes (different grep, empty match, etc.), could be empty and you’d print .
Suggestion:
or only print the suffix when non-empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on PR #452:
URL: https://github.com/apache/hugegraph-doc/pull/452#issuecomment-3956549101
@bitflicker64 Hi, still found some 404 links & broken image link:
- https://hugegraph.apache.org/docs/introduction/README (both in EN & CN)
- https://hugegraph.apache.org/docs/quickstart/hugegraph-ai/quick_start/
<img width="818" height="266" alt="image" src="https://github.com/user-attachments/assets/afedae78-4fe5-4063-880d-43c38ae32106" />
<img width="901" height="320" alt="image" src="https://github.com/user-attachments/assets/715d1e1d-041d-401e-92ab-cb0c18153a81" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/hugegraph-doc/pull/452#issuecomment-3958847028
> BTW, already fixed the first issue in [b1fccc8](https://github.com/apache/hugegraph-doc/commit/b1fccc8faf7e424e1e16d059295d67e84ec726f1)
🫡
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2791588145
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
Review Comment:
P2: `tar.gz` in this regex is interpreted as `tar` + any-char + `gz`, so values like `foo.tarXgz` also match as assets. Escaping the dot (`tar\.gz`) avoids false matches and keeps extension classification precise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786431831
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
+
+ case "$clean_lower" in
+ *.png|*.jpg|*.jpeg|*.svg|*.gif|*.xml|*.yaml|*.yml|*.json|*.css|*.js|*.pdf|*.zip|*.tar.gz)
+ if [[ -f "$target_path" ]]; then
+ return 0
+ else
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+ return
+ fi
+ ;;
+ esac
+
+ if [[ -f "$target_path" || -f "$target_path.md" || -f "$target_path/_index.md" || -f "$target_path/README.md" ]]; then
+ return 0
+ fi
+
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+}
+
+echo "Starting link validation..."
+
+while read -r FILE; do
+ declare -A CODE_LINES
+ in_fence=false
+ line_no=0
+
+ while IFS= read -r line; do
+ ((line_no++))
+
+ if [[ "$line" =~ ^[[:space:]]*(\`\`\`|~~~) ]]; then
+ if $in_fence; then
+ in_fence=false
+ else
+ in_fence=true
+ fi
+ CODE_LINES[$line_no]=1
continue
fi
- # Check for file existence variations
- FOUND=false
-
- # Check 1: As .md file
- if [[ -f "${TARGET_PATH}.md" ]]; then
- FOUND=true
- # Check 2: Exact file (if extension was included)
- elif [[ -f "$TARGET_PATH" ]]; then
- FOUND=true
- # Check 3: Directory index
- elif [[ -f "${TARGET_PATH}/_index.md" ]]; then
- FOUND=true
- # Check 4: Directory README (legacy)
- elif [[ -f "${TARGET_PATH}/README.md" ]]; then
- FOUND=true
+ if $in_fence; then
+ CODE_LINES[$line_no]=1
+ continue
Review Comment:
**‼️ Critical: Inline code detection logic is fragile**
The backtick count parity check (`inline_count % 2 == 1`) assumes no escaped backticks and perfect pairing within a single line. This could produce false positives/negatives with malformed markdown or edge cases like:
- Escaped backticks: \`not code\`
- Multiple inline code spans: `code1` text `code2`
Consider documenting this limitation or using a more robust state-machine approach for inline code detection.
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
Review Comment:
**‼️ Critical: Absolute path mapping hardcoded to English content**
The current logic maps `/*` → `content/en/*`, which causes issues:
1. **Language mismatch**: Chinese docs linking to `/community/maturity/` incorrectly resolve to English paths
2. **Hugo runtime path ambiguity**: Links like `/clients/...` are Hugo URL paths, not filesystem paths. They should map to `content/en/docs/clients/...` but currently map to `content/en/clients/...`
**Current usage analysis**: Only 5 such links exist in the entire codebase:
- `/clients/gremlin-console.html`
- `/community/maturity/`
- `/blog/2025/09/30/...`
- `/blog/2025/10/09/...`
- `/language/hugegraph-gremlin.md`
**Recommended solution (Option B - Skip validation)**:
Since these are Hugo runtime URL paths (not direct filesystem mappings) and only 5 exist, suggest skipping validation for ambiguous absolute paths with a warning:
```bash
elif [[ "$clean_link" == /* ]]; then
# Skip validation for ambiguous absolute paths (Hugo runtime URLs)
echo "Warning: Skipping validation for ambiguous absolute path: $link in $file:$line_no"
echo " (These Hugo runtime paths require manual verification)"
return 0
fi
```
This avoids complex Hugo routing logic while being transparent about what's not checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786610592
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
+
+ case "$clean_lower" in
+ *.png|*.jpg|*.jpeg|*.svg|*.gif|*.xml|*.yaml|*.yml|*.json|*.css|*.js|*.pdf|*.zip|*.tar.gz)
+ if [[ -f "$target_path" ]]; then
+ return 0
+ else
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+ return
+ fi
+ ;;
+ esac
+
+ if [[ -f "$target_path" || -f "$target_path.md" || -f "$target_path/_index.md" || -f "$target_path/README.md" ]]; then
+ return 0
+ fi
+
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+}
+
+echo "Starting link validation..."
+
+while read -r FILE; do
+ declare -A CODE_LINES
+ in_fence=false
+ line_no=0
Review Comment:
I’ve documented this limitation inline to make the behavior explicit rather than adding complex parsing logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#issuecomment-3876088801
sorry for committing the batch ill fix the issues and continue the url fixes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786550744
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
Review Comment:
I’ve updated the logic to explicitly skip validation for these cases with a warning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#issuecomment-3882777103
Thanks for the review. I will debug the issues and push a proper fix after my classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2791701354
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
Review Comment:
I’ve added set -o errexit and set -o pipefail at the start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786435964
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
+
+ case "$clean_lower" in
+ *.png|*.jpg|*.jpeg|*.svg|*.gif|*.xml|*.yaml|*.yml|*.json|*.css|*.js|*.pdf|*.zip|*.tar.gz)
+ if [[ -f "$target_path" ]]; then
+ return 0
+ else
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+ return
+ fi
+ ;;
+ esac
+
+ if [[ -f "$target_path" || -f "$target_path.md" || -f "$target_path/_index.md" || -f "$target_path/README.md" ]]; then
+ return 0
+ fi
+
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+}
+
+echo "Starting link validation..."
+
+while read -r FILE; do
+ declare -A CODE_LINES
+ in_fence=false
+ line_no=0
Review Comment:
**🧹 Minor: Code fence detection doesn't validate pairing**
The regex `^[[:space:]]*(\`\`\`|~~~)` allows leading spaces but doesn't track whether fences are properly paired. If opening and closing fences are mismatched, all subsequent content may be incorrectly treated as code.
Edge case example:
```markdown
\`\`\`bash
code here
# Missing closing fence - everything below treated as code
[This link won't be checked](broken-link.md)
```
Consider tracking fence state more carefully or documenting this limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786436948
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Review Comment:
**🧹 Minor: Missing error handling for directory resolution**
The script assumes all `cd` operations succeed when resolving `SCRIPT_DIR`, `REPO_ROOT`, and `CONTENT_ROOT`. If any `cd` fails, subsequent paths will be incorrect but the script continues.
Suggest adding error handling:
```suggestion
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" || exit 1
CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)" || exit 1
```
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
Review Comment:
**🧹 Minor: No verbose mode for debugging**
External links, Hugo shortcodes, and code blocks are silently skipped without any output. This makes debugging difficult when trying to understand why a specific link wasn't validated.
Consider adding a verbose mode (e.g., `-v` flag or `VERBOSE=1` env var) to report:
- Skipped external links
- Skipped Hugo shortcodes (`{{`...)
- Links within code blocks
- Which validation path was taken for each link
This would help maintainers understand the script's behavior without modifying it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#issuecomment-3873555808
I improved the link validator by adding canonical path resolution, content directory boundary checks, and accurate file line reporting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786591679
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
Review Comment:
I’ve moved the asset extension list into a top level variable so it’s easier to maintain, and added the commonly used image, font, and media formats you mentioned. This should make it more flexible going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2791600797
##########
content/en/docs/clients/restful-api/rebuild.md:
##########
@@ -82,4 +82,4 @@ PUT http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/jobs/rebuild/edge
Note:
-> You can get the asynchronous job status by `GET http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/tasks/${task_id}` (the task_id here should be 3). See More [AsyncJob RESTfull API](../task)
\ No newline at end of file
+> You can get the asynchronous job status by `GET http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/tasks/${task_id}` (the task_id here should be 3). See More [AsyncJob RESTfull API](./task)
Review Comment:
Typo in link text: “RESTfull” should be “RESTful”.
##########
content/en/docs/clients/restful-api/rebuild.md:
##########
@@ -30,7 +30,7 @@ PUT http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/jobs/rebuild/inde
```
Note:
-> You can get the asynchronous job status by `GET http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/tasks/${task_id}` (the task_id here should be 1). See More [AsyncJob RESTfull API](../task)
+> You can get the asynchronous job status by `GET http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/tasks/${task_id}` (the task_id here should be 1). See More [AsyncJob RESTfull API](./task)
Review Comment:
Typo in link text: “RESTfull” should be “RESTful”.
##########
content/en/docs/clients/restful-api/rebuild.md:
##########
@@ -56,7 +56,7 @@ PUT http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/jobs/rebuild/vert
Note:
-> You can get the asynchronous job status by `GET http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/tasks/${task_id}` (the task_id here should be 2). See More [AsyncJob RESTfull API](../task)
+> You can get the asynchronous job status by `GET http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/tasks/${task_id}` (the task_id here should be 2). See More [AsyncJob RESTfull API](./task)
Review Comment:
Typo in link text: “RESTfull” should be “RESTful”.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2793512035
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,283 @@
#!/bin/bash
+set -o errexit
+set -o pipefail
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ if [[ "$VERBOSE" == "1" ]]; then
+ echo "Info: $*"
+ fi
+}
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar\.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" || exit 1
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)" || exit 1
+
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ # Decode common URL-encoded characters explicitly
+ link="${link//%20/ }" # space
+ link="${link//%23/#}" # hash
+ link="${link//%2F/\/}" # forward slash
+
+ # Generic percent-decoding for remaining cases
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+ local parts
+
+ # Bash 3.2 compatible: use here-string
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ # Bash 3.2 compatible: calculate last index instead of using -1
+ if [[ ${#result[@]} -gt 0 ]]; then
+ local last_idx=$((${#result[@]} - 1))
+ unset "result[$last_idx]"
+ fi
+ else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use Python to compute realpath which resolves symlinks AND normalizes paths
+ # Python's os.path.realpath is tolerant of non-existent final targets
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback: Normalize without symlink resolution if Python3 unavailable
+ # Note: This won't resolve symlinks, only normalize .. and . components
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ log_verbose "Skipping Hugo shortcode link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ local clean_lower
+ clean_lower="$(printf "%s" "$clean_link" | tr '[:upper:]' '[:lower:]')"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ log_verbose "Skipping external link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+
+ elif [[ "$clean_link" == /community/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
-
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
-
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
-
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ elif [[ "$clean_link" == /blog/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+
+ elif [[ "$clean_link" == /language/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+
+ elif [[ "$clean_link" == /clients/* ]]; then
+ target_path="$REPO_ROOT/static${clean_link}"
+
+ elif [[ "$clean_link" == /* ]]; then
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Unsupported absolute internal path (cannot validate deterministically)"
+ echo " File: $location"
+ echo " Link: $link"
+
+ EXIT_CODE=1
+ return
+
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ "$REPO_ROOT/static"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
+
+ if [[ "$clean_lower" =~ \.(${ASSET_EXTENSIONS_REGEX})$ ]]; then
+ if [[ -f "$target_path" ]]; then
+ return 0
else
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+ return
+ fi
+ fi
+
+ if [[ -f "$target_path" || -f "$target_path.md" || -f "$target_path/_index.md" || -f "$target_path/README.md" ]]; then
+ return 0
+ fi
+
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+}
+
+echo "Starting link validation..."
+
+while IFS= read -r FILE; do
+
+ CODE_LINES=""
+ in_fence=false
+ line_no=0
+
+ while IFS= read -r line || [[ -n "$line" ]]; do
+ ((++line_no))
+ # NOTE:
+ # Code fence detection is heuristic and does not validate proper pairing.
+ # The logic simply toggles state when encountering ``` or ~~~ markers.
+ # If a Markdown file contains an unclosed fence or mismatched fence types,
+ # all subsequent lines may be treated as code and skipped from validation.
+ # This behavior is intentional to keep the validator lightweight and
+ # avoids implementing a full Markdown parser. Such cases require manual review.
+ if [[ "$line" =~ ^[[:space:]]*(\`\`\`|~~~) ]]; then
+ # NOTE:
+ # Code fence detection assumes fences are properly paired.
+ # If a Markdown file contains an unclosed or mismatched fence,
+ # subsequent content may be treated as code and skipped.
+ # This script does not attempt full Markdown validation.
+
+ if $in_fence; then
+ in_fence=false
+ else
+ in_fence=true
+ fi
+ CODE_LINES="$CODE_LINES $line_no "
continue
fi
- # Check for file existence variations
- FOUND=false
-
- # Check 1: As .md file
- if [[ -f "${TARGET_PATH}.md" ]]; then
- FOUND=true
- # Check 2: Exact file (if extension was included)
- elif [[ -f "$TARGET_PATH" ]]; then
- FOUND=true
- # Check 3: Directory index
- elif [[ -f "${TARGET_PATH}/_index.md" ]]; then
- FOUND=true
- # Check 4: Directory README (legacy)
- elif [[ -f "${TARGET_PATH}/README.md" ]]; then
- FOUND=true
+ if $in_fence; then
+ CODE_LINES="$CODE_LINES $line_no "
+ continue
fi
- if [ "$FOUND" = false ]; then
- echo "Error: Broken link in $FILE"
- echo " Link: $LINK"
- echo " Target: $TARGET_PATH (and variants)"
- EXIT_CODE=1
+ # NOTE:
+ # Inline code detection is heuristic and intentionally simplistic.
+ # The logic assumes backticks are properly paired within a single line
+ # after removing escaped backticks. Malformed Markdown, complex inline
+ # constructs, or unusual escaping patterns may cause false positives
+ # or false negatives. This validator does not implement a full Markdown
+ # parser and therefore cannot guarantee perfect inline code detection.
+ escaped_line="${line//\\\`/}"
+ inline_count=$(printf "%s\n" "$escaped_line" | grep -o "\`" || true)
+ inline_count=$(printf "%s\n" "$inline_count" | wc -l)
+ if (( inline_count % 2 == 1 )); then
Review Comment:
‼️ **Potential false negative in inline-code detection**: lines without backticks are currently treated as if they had an odd backtick count, so non-code lines are added to `CODE_LINES` and skipped. This can allow broken links to pass CI undetected. Could we normalize the `wc -l` output before parity check?
```suggestion
escaped_line="${line//\\\`/}"
inline_count=$(printf "%s\n" "$escaped_line" | grep -o "\`" | wc -l || true)
inline_count="${inline_count//[[:space:]]/}"
if (( inline_count % 2 == 1 )); then
CODE_LINES="$CODE_LINES $line_no "
fi
```
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,283 @@
#!/bin/bash
+set -o errexit
+set -o pipefail
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ if [[ "$VERBOSE" == "1" ]]; then
+ echo "Info: $*"
+ fi
+}
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar\.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" || exit 1
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)" || exit 1
+
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ # Decode common URL-encoded characters explicitly
+ link="${link//%20/ }" # space
+ link="${link//%23/#}" # hash
+ link="${link//%2F/\/}" # forward slash
+
+ # Generic percent-decoding for remaining cases
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+ local parts
+
+ # Bash 3.2 compatible: use here-string
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ # Bash 3.2 compatible: calculate last index instead of using -1
+ if [[ ${#result[@]} -gt 0 ]]; then
+ local last_idx=$((${#result[@]} - 1))
+ unset "result[$last_idx]"
+ fi
+ else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use Python to compute realpath which resolves symlinks AND normalizes paths
+ # Python's os.path.realpath is tolerant of non-existent final targets
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback: Normalize without symlink resolution if Python3 unavailable
+ # Note: This won't resolve symlinks, only normalize .. and . components
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ log_verbose "Skipping Hugo shortcode link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ local clean_lower
+ clean_lower="$(printf "%s" "$clean_link" | tr '[:upper:]' '[:lower:]')"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ log_verbose "Skipping external link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+
+ elif [[ "$clean_link" == /community/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
-
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
-
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
-
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ elif [[ "$clean_link" == /blog/* ]]; then
Review Comment:
‼️ **`/blog/*` and `/cn/blog/*` are permalink-based routes** and usually do not map 1:1 to `content/...` filesystem paths. Valid links can be reported as broken (or unsupported) even when Hugo renders them correctly. Suggest skipping deterministic file checks for these routes to avoid false positives.
```suggestion
elif [[ "$clean_link" == /blog/* || "$clean_link" == /cn/blog/* ]]; then
# Blog URLs are permalink-based and don't map 1:1 to content file paths.
# Skip deterministic filesystem validation for these routes.
log_verbose "Skipping permalink-based blog link: $link ($file:$line_no)"
return 0
```
##########
content/en/docs/SUMMARY.md:
##########
@@ -1,69 +1,67 @@
# HugeGraph Docs
-* [Download](download.md)
+* [Download](download/download)
## Quickstart
-* [Install HugeGraph-Server](quickstart/hugegraph-server.md)
-* [Load data with HugeGraph-Loader](quickstart/hugegraph-loader.md)
-* [Visual with HugeGraph-Hubble](quickstart/hugegraph-hubble.md)
-* [Develop with HugeGraph-Client](quickstart/hugegraph-client.md)
-* [Manage with HugeGraph-Tools](quickstart/hugegraph-tools.md)
-* [Analysis with HugeGraph-Computer](quickstart/hugegraph-computer.md)
-* [Display with HugeGraph-Studio](quickstart/hugegraph-studio.md)
+* [Install HugeGraph-Server](quickstart/hugegraph/hugegraph-server)
+* [Load data with HugeGraph-Loader](quickstart/toolchain/hugegraph-loader)
+* [Visual with HugeGraph-Hubble](quickstart/toolchain/hugegraph-hubble)
+* [Develop with HugeGraph-Client](quickstart/client/hugegraph-client)
+* [Manage with HugeGraph-Tools](quickstart/toolchain/hugegraph-tools)
+* [Analysis with HugeGraph-Computer](quickstart/computing/hugegraph-computer)
+* [Display with HugeGraph-Studio](quickstart/hugegraph-studio)
Review Comment:
⚠️ This link currently resolves to a 404 in rendered docs (`/docs/quickstart/hugegraph-studio/`, and same issue for the CN summary entry). Since this PR updates `SUMMARY.md` links for correctness, could we also update or remove this entry to keep navigation valid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#issuecomment-3875909329
> If the total number of changed lines isn't too large, I'd suggest fixing them together in this PR. Otherwise, (Required) CI failures might be troublesome & blocking the merge, and these fixes should have minimal side effects.
ill start working on fixing the links
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786570707
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
Review Comment:
Added clear decoding for common URL-encoded characters before the general decode step the generic handling remains for everything else
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#issuecomment-3877876735
Link validation checks are passing now. Fixed broken relative links, corrected SUMMARY navigation for Hugo routing, and removed an invalid reference to a nonexistent file Remaining messages are expected Hugo runtime URL warnings only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on PR #452:
URL: https://github.com/apache/hugegraph-doc/pull/452#issuecomment-3957445258
> Makes sense those are absolute [hugegraph.apache.org](https://hugegraph.apache.org/)... links, and the validator currently skips all http/https targets, so it won’t catch site 404s. We can either update the docs to use /docs/... (so they’re checked), or add an optional allowlist-based external check for hugegraph.apache.org. Would you prefer I add an optional allowlist , or keep CI limited to internal links and handle these in follow up fixes?
We could unify the link path if possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#issuecomment-3875860375
If the total number of changed lines isn't too large, I'd suggest fixing them together in this PR. Otherwise, CI failures might be troublesome, and these fixes should have minimal side effects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786433953
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
Review Comment:
**⚠️ Important: URL decoding might not handle all edge cases**
The current generic decoding approach may not handle all URL-encoded characters correctly. Consider handling common cases explicitly before the generic decode:
```suggestion
# Decode common URL-encoded characters explicitly
link="${link//%20/ }" # space
link="${link//%23/#}" # hash
link="${link//%2F/\/}" # forward slash
# Then do generic decoding for remaining cases
link="${link//%/\\x}"
link="$(printf '%b' "$link")"
```
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
Review Comment:
**⚠️ Important: Asset file extension whitelist may be incomplete**
Current list: `.png|.jpg|.jpeg|.svg|.gif|.xml|.yaml|.yml|.json|.css|.js|.pdf|.zip|.tar.gz`
Consider adding commonly used formats:
- Modern image formats: `.webp`, `.avif`, `.ico`
- Web fonts: `.woff`, `.woff2`, `.ttf`, `.eot`
- Other assets: `.mp4`, `.webm` (if videos are used)
Alternatively, make this list configurable via a variable at the top of the script for easier maintenance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2791699623
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,249 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
+VERBOSE="${VERBOSE:-0}"
+
+log_verbose() {
+ [[ "$VERBOSE" == "1" ]] && echo "Info: $*"
+}
+
+
+ASSET_EXTENSIONS_REGEX='png|jpg|jpeg|svg|gif|webp|avif|ico|xml|yaml|yml|json|css|js|pdf|zip|tar.gz|woff|woff2|ttf|eot|mp4|webm'
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 1
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" || exit 1
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)" || exit 1
+
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ # Decode common URL-encoded characters explicitly
+ link="${link//%20/ }" # space
+ link="${link//%23/#}" # hash
+ link="${link//%2F/\/}" # forward slash
+
+ # Generic percent-decoding for remaining cases
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
+
+ link="${link%%#*}"
+ link="${link%%\?*}"
+
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
+
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
+ else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use Python to compute realpath which resolves symlinks AND normalizes paths
+ # Python's os.path.realpath is tolerant of non-existent final targets
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback: Normalize without symlink resolution if Python3 unavailable
+ # Note: This won't resolve symlinks, only normalize .. and . components
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ log_verbose "Skipping Hugo shortcode link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ local clean_lower="${clean_link,,}"
+
+ if [[ "$clean_lower" == http://* || "$clean_lower" == https://* || "$clean_lower" == "//"* ]]; then
+ log_verbose "Skipping external link: $link ($file:$line_no)"
+ return 0
+ fi
+
+ case "$clean_lower" in
+ mailto:*|tel:*|javascript:*|data:*)
+ return 0
+ ;;
+ esac
+
+ if [[ "$clean_link" == /docs/* ]]; then
+ target_path="$CONTENT_ROOT/en${clean_link}"
+ elif [[ "$clean_link" == /cn/docs/* ]]; then
+ target_path="$CONTENT_ROOT${clean_link}"
+ elif [[ "$clean_link" == /* ]]; then
+ # Skip validation for ambiguous absolute paths (Hugo runtime URLs)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Warning: Skipping validation for ambiguous absolute path"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Reason: Hugo runtime URL (not directly mappable to filesystem)"
+ return 0
+ else
+ local file_dir
+ file_dir="$(cd "$(dirname "$file")" && pwd)"
+ target_path="$file_dir/$clean_link"
+ fi
+
+ target_path="$(canonicalize_path "$target_path")"
+ target_path="$(resolve_real_path "$target_path")"
+
+ case "$target_path" in
+ "$CONTENT_ROOT"/*) ;;
+ *)
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Link resolves outside content directory"
+ echo " File: $location"
+ echo " Link: $link"
+ EXIT_CODE=1
+ return
+ ;;
+ esac
+
+ if [[ "$clean_lower" =~ \.(${ASSET_EXTENSIONS_REGEX})$ ]]; then
+ if [[ -f "$target_path" ]]; then
+ return 0
+ else
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+ return
+ fi
+ fi
+
+ if [[ -f "$target_path" || -f "$target_path.md" || -f "$target_path/_index.md" || -f "$target_path/README.md" ]]; then
+ return 0
+ fi
+
+ location="$file"
+ [[ -n "$line_no" ]] && location="$file:$line_no"
+
+ echo "Error: Broken link"
+ echo " File: $location"
+ echo " Link: $link"
+ echo " Target: $target_path"
+ EXIT_CODE=1
+}
+
echo "Starting link validation..."
-# Find all markdown files and verify links
while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+ declare -A CODE_LINES
Review Comment:
Bash 4 dependency has been removed in the latest revision . So enforcing Bash 4+ should no longer be necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on code in PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#discussion_r2786669128
##########
dist/validate-links.sh:
##########
@@ -1,63 +1,213 @@
#!/bin/bash
-# Configuration
CONTENT_DIR="content"
EXIT_CODE=0
-echo "Starting link validation..."
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+CONTENT_ROOT="$(cd "$REPO_ROOT/$CONTENT_DIR" && pwd)"
-# Find all markdown files and verify links
-while read -r FILE; do
- # Extract internal links starting with /docs/ or /cn/docs/
- # We look for [text](url) pattern where url starts with /docs/ or /cn/docs/
- # Using grep to find all matching links in the file
- while read -r MATCH; do
- if [ -z "$MATCH" ]; then continue; fi
+if [[ ! -d "$CONTENT_ROOT" ]]; then
+ echo "Error: content directory not found. Run from repository root."
+ exit 1
+fi
+
+normalize_link() {
+ local link="$1"
+
+ link="${link//%/\\x}"
+ link="$(printf '%b' "$link")"
- # Extract URL from ](url)
- LINK=${MATCH#*](}
- LINK=${LINK%)}
+ link="${link%%#*}"
+ link="${link%%\?*}"
- # Remove anchor and query parameters
- CLEAN_LINK=$(echo "$LINK" | cut -d'#' -f1 | cut -d'?' -f1)
- CLEAN_LINK=${CLEAN_LINK%/}
+ if [[ "$link" != "/" ]]; then
+ link="${link%/}"
+ fi
- # Determine target file path based on language prefix
- if [[ "$CLEAN_LINK" == /docs/* ]]; then
- TARGET_PATH="content/en${CLEAN_LINK}"
- elif [[ "$CLEAN_LINK" == /cn/docs/* ]]; then
- TARGET_PATH="content${CLEAN_LINK}"
+ printf "%s" "$link"
+}
+
+canonicalize_path() {
+ local path="$1"
+ local result=()
+ local part
+
+ IFS='/' read -r -a parts <<< "$path"
+
+ for part in "${parts[@]}"; do
+ if [[ -z "$part" || "$part" == "." ]]; then
+ continue
+ elif [[ "$part" == ".." ]]; then
+ if [[ ${#result[@]} -gt 0 ]]; then
+ unset 'result[-1]'
+ fi
else
+ result+=("$part")
+ fi
+ done
+
+ if [[ ${#result[@]} -eq 0 ]]; then
+ printf "/"
+ else
+ ( IFS='/'; printf "/%s" "${result[*]}" )
+ fi
+}
+
+resolve_real_path() {
+ local path="$1"
+
+ if command -v python3 >/dev/null 2>&1; then
+ # Use python to compute realpath which is tolerant of non existing final target
+ python3 - <<'PY' "$path"
+import os
+import sys
+p = sys.argv[1]
+# os.path.realpath resolves symlinks for existing components and otherwise returns a normalized path
+print(os.path.realpath(p))
+PY
+ else
+ # Fallback to the safe canonicalize_path output if python3 is not available
+ canonicalize_path "$path"
+ fi
+}
+
+check_internal_link() {
+ local link="$1"
+ local file="$2"
+ local line_no="$3"
+ local clean_link
+ local target_path
+ local location
+
+ clean_link="$(normalize_link "$link")"
+
+ [[ -z "$clean_link" || "$clean_link" == "#" ]] && return 0
+
+ if [[ "$clean_link" == "{{"* ]]; then
+ return 0
+ fi
+
Review Comment:
donee
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452#issuecomment-3884063233
All validator issues raised in the review thread have been addressed.
Path normalization, absolute path handling, URL decoding, and asset resolution logic are corrected.
Heuristic behaviors for code fences and inline code detection are now explicitly documented.
The script is fail fast, Bash compatible, and produces deterministic validation results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [incubator-hugegraph-doc]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin merged PR #452:
URL: https://github.com/apache/incubator-hugegraph-doc/pull/452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] Harden documentation link validation to prevent false CI passes [hugegraph-doc]
Posted by "bitflicker64 (via GitHub)" <gi...@apache.org>.
bitflicker64 commented on PR #452:
URL: https://github.com/apache/hugegraph-doc/pull/452#issuecomment-3957400918
> @bitflicker64 Hi, still found some 404 links & broken image link:
>
> * https://hugegraph.apache.org/docs/introduction/README (both in EN & CN)
> * https://hugegraph.apache.org/docs/quickstart/hugegraph-ai/quick_start/
>
Makes sense those are absolute https://hugegraph.apache.org/... links, and the validator currently skips all http/https targets, so it won’t catch site 404s. We can either update the docs to use /docs/... (so they’re checked), or add an optional allowlist-based external check for hugegraph.apache.org. Would you prefer I add an optional allowlist , or keep CI limited to internal links and handle these in follow up fixes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org