[PR] [fix](seo) Keep untranslated blog pages out of the index and publish every locale sitemap [doris-website]

["morningman (via GitHub)" <gi...@apache.org>]

morningman opened a new pull request, #4325:
URL: https://github.com/apache/doris-website/pull/4325

   ## Summary
   
   The blog has no translations: the Chinese ones were removed on purpose in 2023 (#272) and Japanese has not started. The blog plugin falls back per file, so `/zh-CN/blog/<post>/` and `/ja/blog/<post>/` served the English article on an indexable, self-canonical URL, listed in that locale's sitemap and advertised as an `hreflang` alternate. Every article therefore existed two or three times as a page Google may pick, which splits the signals for the article and spends crawl budget on 428 duplicates instead of the pages that really are localized.
   
   At the same time each locale generates its own sitemap, but only the English one was declared anywhere, so 6120 Chinese and 1288 Japanese URLs had no sitemap-based discovery path, and no sitemap carried `lastmod`.
   
   ### 1. Localized blog pages of an untranslated blog
   
   `config/i18n.js` gains `hasBlogTranslations`: a locale counts as translating the blog only when its `docusaurus-plugin-content-blog` directory holds a post file (an empty directory, or one holding just `tags.yml`, translates nothing). The render path reads the resulting locale list through `defaultLocaleBlogLocales`.
   
   - `BlogLayout` marks the whole blog section of such a locale `noindex, follow`. The section stays browsable, and the sitemap plugin drops noindex routes, so those URLs leave that locale's sitemap too. This is the same treatment the docs already give an untranslated page (`DocItem/Metadata`).
   - `BlogPostPage/Metadata` points each of those posts at the default locale's copy with a canonical link, in the URL form the `hreflang` alternates already use.
   
   Both decisions are derived from the filesystem, so translating the blog again — or starting it in Japanese — lifts the gate by itself, and a test records the current intent so the SEO decision is revisited rather than changed by accident.
   
   ### 2. Sitemap discovery and freshness
   
   - `robots.txt` declares `/zh-CN/sitemap.xml` and `/ja/sitemap.xml`. Google retired the sitemap ping endpoint, so robots.txt and Search Console are the only ways a sitemap is found. Both deploy workflows set `DORIS_ENABLE_JA: 'true'`, so the Japanese sitemap is present wherever this file is served.
   - The sitemap carries `lastmod` from each file's last commit, the only field Google still uses for crawl scheduling. `changefreq` and `priority`, which the configuration set, are ignored. This completes the source timestamps `9adabb13c9` kept for sitemap entries.
   - `manual-deploy-website.yml` checks out full history for it; `cron-deploy-website.yml` already did.
   - The utility-route filter in `createSitemapItems` is locale-generic, so `/ja/search` is excluded like its English and Chinese counterparts. The list was hardcoded to `en`/`zh-CN`.
   - The search page's robots directive uses `name=` instead of `property=`, the form Google documents and the form the sitemap plugin reads when it decides a route is noindex.
   
   ## Verification
   
   - `node --test scripts/i18n/__tests__/*.cjs` — 81 pass, including new cases for the detection rules and a guard recording that neither shipped locale translates the blog yet.
   - `node --test scripts/docs-governance/__tests__/*.test.js scripts/__tests__/*.test.js` — 37 pass.
   - The site configuration resolves with and without `ja`, with the expected locale lists and the `lastmod` option set.
   - Rendered in development mode (`--locale ja`): `/ja/blog/<post>/`, `/ja/blog/` and `/ja/blog/tags/` come out `noindex` with the canonical on the English copy; `/ja/download/` and a translated `/ja/docs/4.x` page are untouched, so the gate stays inside the blog.
   
   The sitemap contents and hreflang effects of the built site are for the build and deploy checks to confirm.
   
   ## Not in this change
   
   - `/zh-CN/course/...`, `/zh-CN/why-doris/...` and `/zh-CN/use-cases/...` still serve English pages on indexable Chinese URLs. Those are untranslated pages rather than a fallback rule like the blog's, and `config/i18n.js` declares Chinese ordinary pages translated, so deciding their indexability is a product call. Translating them, or declaring them per page, is the follow-up.
   - `htmlLang` and therefore `hreflang` stay `en-US` / `zh-Hans-CN`. Narrowing those to `en` / `zh-CN` is a separate discussion.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [PR] [fix](i18n) De-index untranslated blog pages, publish every locale sitemap, and fix localized heading IDs and search [doris-website]

["morningman (via GitHub)" <gi...@apache.org>]

morningman merged PR #4325:
URL: https://github.com/apache/doris-website/pull/4325


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org