[GitHub] [airflow] uranusjr commented on a change in pull request #17946: Add robots.txt and X-Robots-Tag header

[GitBox <gi...@apache.org>]

uranusjr commented on a change in pull request #17946:
URL: https://github.com/apache/airflow/pull/17946#discussion_r700004826



##########
File path: airflow/www/views.py
##########
@@ -2924,6 +2925,16 @@ def tree_data(self):
         # avoid spaces to reduce payload size
         return htmlsafe_json_dumps(tree_data, separators=(',', ':'))
 
+    @expose('/robots.txt')
+    @action_logging
+    def robots(self):
+        """
+        Returns a robots.txt file for blocking certain search engine crawlers. This mitigates some
+        of the risk associated with exposing Airflow to the public internet, however it does not
+        address the real security risks associated with such a deployment.
+        """
+        return send_from_directory(current_app.static_folder, 'robots.txt')

Review comment:
       For such a simple file, it’s probably easier to simply embed `robot.txt` in the view function instead of using `send_from_directory` (which introduces filesystem overhead).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org