[GitHub] [pulsar-site] tisonkun commented on a diff in pull request #591: In order to ensure that the api documentation is up-to-date in search…

["tisonkun (via GitHub)" <gi...@apache.org>]

tisonkun commented on code in PR #591:
URL: https://github.com/apache/pulsar-site/pull/591#discussion_r1221118072


##########
static/robots.txt:
##########
@@ -1 +1,5 @@
 Sitemap: https://pulsar.apache.org/sitemap.xml
+User-agent: *
+Allow: /
+Disallow: /api/

Review Comment:
   We should exclude legacy versions only. Maintained versions should be still indexed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org