[GitHub] [airflow] thejens commented on a change in pull request #17946: Add robots.txt and X-Robots-Tag header

[GitBox <gi...@apache.org>]

thejens commented on a change in pull request #17946:
URL: https://github.com/apache/airflow/pull/17946#discussion_r700014174



##########
File path: airflow/www/views.py
##########
@@ -2924,6 +2925,16 @@ def tree_data(self):
         # avoid spaces to reduce payload size
         return htmlsafe_json_dumps(tree_data, separators=(',', ':'))
 
+    @expose('/robots.txt')
+    @action_logging
+    def robots(self):
+        """
+        Returns a robots.txt file for blocking certain search engine crawlers. This mitigates some
+        of the risk associated with exposing Airflow to the public internet, however it does not
+        address the real security risks associated with such a deployment.
+        """
+        return send_from_directory(current_app.static_folder, 'robots.txt')

Review comment:
       easy enough to change, just thought it'd be easier to maintain as a physical file. - wouldn't expect much traffic/load on this endpoint anyway




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org